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ABSTRACT 



Three separate projects were involved in this 
evaluation project. Project 1 involved teacher- part icipants, teaching 
staffs, and institute directors and asked their opinions about 
general institute performance and particular subject matter coverage. 
Project 2 focused on the teacher-participants only, investigating the 
effects of institute attendance on 1) their learning of subject 
matter content, 2) their attitudes toward students, and 3) their 
consideration of teaching strategies. Project 3 concentrated on the 
host colleges and universities, specifically seeking to explore some 
of the reactions and experiences affecting key educators. A sample of 
94 institutes was involved, consisting of 30 in history, 13 in 
political science, 24 in educational media, and 27 in education of 
the disadvantaged. The total sample of teachers exceeded 3,000. The 
general consensus of comments by directors, staff members, and 
participants was favorable, many stating that the Institutes had 
effected some changes in their consideration and education of 
teachers although it was not possible to show statistically 
significant pre- and post-institute differences with regard to 
improved subject matter achievements. (MBM) 
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FOREWORD 



The Final Report , PROJECT IMPACTS A Pilot Study Evaluating 
the NDEA Summer Institute Program was prepared and submitted from 
the American Institutes for Research, Pittsburgh, Pennsylvania, 
by Evelyn Perloff, Clyde F. Kohn * and Robert D. Gatewood, 

This document reports the research undertaken on a pilot 
basis during the period 1966 - 1970 in an effort to provide an 
evaluation of the impact of the summer institute program outlined 
by Title XI of the National Defense Education Act of 1958. 

The PROJECT was a large-scale, pioneer effort to obtain data 
which would aid in arriving at an evidence-based judgment concerning 
the effectiveness of the program. Since it was an innovative effort, 
there were few guidelines from previous similar research that could 
be used as a foundation for designing the investigation. Accordingly, 
the PROJECT personnel were faced with the challenging task of 
inventing appropriate procedures on a pilot basis. The PROJECT 
personnel are to be congratulated that their research turned out to 
be as fruitful as it was. 

The strategy was to mount a three-pronged attack on the 
basic question of the effectiveness of the institute program, ** 
Project 1, focussing on participants and institute staff, assessed 
attitudes and opinions concerning the effectiveness of the institutes. 
Project 2 sought to measure pre-to-post institute changes in the 
participants. Project 3 was intended to assess the influence of 
the institutes on the host academic institutions. 

The report presents a wealth of detail on all three projects. 
Each of the three component parts is considered separately, since 
each is different in substance and methodology. 



* University of Iowa 

** The essence of the report is contained in the Summary , pages 
1-5, and the Conclusions , pages 81 - 82. 
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PROJECT IMPACT and its outcomes attest to the difficulties 
inherent in empirical research in real-life educational settings. 

The personnel of the PROJECT have done a fine job, and a courageous 
one. That their efforts were not more successful than they were 
should in no way reflect upon them; indeed, it is encouraging 
that as much information was yielded by PROJECT IMPACT as is 
contained in the report, which includes findings, observations, and 
suggestions that will be valuable to people involved in the design, 
planning, and day-to-day administration of future short-term institutes. 

In sum, although technical difficulties made it impossible 
to demonstrate clear pre-to-post changes produced by the institutes, 
the institute participants, instructors, and administrators 
generally evaluated the institutes and their effectiveness positively. 
The large majority of people who have been involved in the NDEA 
Summer Institute Program believe that the institutes have improved 
the performance, knowledge, and perspective of the teachers who have 
had the opportunity to benefit from them, and that the Program 
has aided in a paramount contemporary social endeavor: the continual 
improvement of the teaching in our nation’s schools. 

Accordingly, the CONPASS Board conveys its thanks to the 
PROJECT IMPACT personnel for a difficult task taken on and 
accomplished with imagination and courage. 



The Consortium of Professional Associations 
for Study of Special Teacher Improvement Programs 

December 7, 1970 
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IN SUMMARY 



The major purpose of PROJECT IMPACT was to evaluate the educational 
impact of the summer institute program, as outlined by Title Xi of the 
National Defense Education Act of 195ft, upon the teacher-participants who 
attended and the colleges and universities that hosted the institutes. 

Under Title XI, programs were established to improve the quality of teaching 
and to provide training for educational personnel in order to help meet 
critical shortages in numerous disciplines. To accomplish these goals, the 
now well-accepted summer institute plan which has provided unparalleled 
training opportunities for thousands of school teachers was organized. At 
the same time, hundreds of college professors of diverse disciplines were 
brought into the business of training school teachers, heretofore regarded 
as taboo. 

PROJECT IMPACT consisted of three separate projects, each describing a 
distinctly different aspect of the impact of the NDEA summer institute pro- 
gram. Project 1, involving teacher-participants, teaching staffs, and insti- 
tute directors, asked their opinions relative to general institute perfor- 
mance and particular subject matter coverage. Project 2 focused on the 
teacher-participants only, investigating the effects of institute attendance 
on the following: (1) their learning of subject matter content; (2) their 

attitudes toward students; and, (3) their consideration of teaching strate- 
gies. Project 3 concentrated on colleges and un i vers i t i es wh i ch hosted these 
summer institutes, specifically seeking to explore some of the reactions and 
experiences affecting key educators at these educational inst i tut ions. 

Typically, summer institutes enrolled about 40 elementary or secondary 
school teachers who were given daily instruction for six to eight weeks. 

Two or three courses were generally offered in addition to a "workshop" 
covering discussions of instructional methods and materials. The institutes 
often concentrated on a common subject matter theme, a set of general prob- 
lems, or addressed themselves to articulating some of the inst ruct iona 1 
strategies specific to a discipline. For the most part, the courses were 
organized to present new material to teachers who had already been well- 
trained, instead of concentrating on remedial upgrading of poorly prepared 
teachers. 

Since there were 12 different areas covering 493 institutes involving 
some 20,000 participants, it was clear that decisions had to be made for 
studying only a few of these to represent the NDEA summer program of 1967. 

Four areas were, therefore, selected to constitute the sample: educational 

media (the "hardware 11 components), history (the area with an earlier follow- 
up), political science (the most recent institute additions), and instruc- 
tion for teachers of disadvantaged youth (the area for greatest potential 
at t i tude change) . 

This plan resulted in a sample of 94 institutes, and consisted of 30 in 
history, 13 in political science, 24 in educational media, and 27 for teachers 
of the disadvantaged. Since approximately 30 to 60 participants were accepted 
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by an institute, the total sample of teachers enrolled in the 3k institutes 
came to more than 3,000. With this relatively large number of subjects 
available, we felt confident that we would be able to draw upon a sample 
adequately representative for the development of educationally sound infer- 
ences vis-a-vis the institute program and its values to the teacher- 
participant . 



PROJECT 1 



Project 1 was the most general in nature, reflecting the objectives 
and preferences of a majority of institute participants and staffs. This 
study considered what actually took place at our sample of 3k institutes. 

Our plans centered on trying to relate stated summer institute objectives 
to actual activities offered at the institute. We wished to learn how much 
emphasis each institute gave to a variety of teaching objectives. 

Since visits to each institute were not possible, we developed a 
questionnaire, the Survey of Institute Objectives, for obtaining the reac- 
tions of various members of the summer institutes — participants, staff mem- 
bers., and directors. Each of the four Surveys of Institute Objectives, 
one for each subject area, was a 50 to 60 item questionnaire dealing with 
the amount of attention placed on certain important objectives by a summer 
institute in that subject matter. Alternatives indicating how much emphasis 
was given varied from “none" to "very much" and all respondents (teachers, 
staff members, and directors) were asked to check the alternative which most 
nearly described their reactions to each statement. A second section of the 
Survey asked respondents to evaluate personally the importance of each of 
the major categories. That is, regardless of the attention given to these 
major topics by their institutes, the respondents were asked which categories 
of objectives they MOST and LEAST preferred. 

The Surveys were scored by assigning one point for an answer of "none" 
to five points for an answer of "very much." Average scores were computed 
for both objectives and preferences, permitting us to compare the interests 
of respondents wi th the actual activities at the institutes. 

Analysis of the Survey of Institute Objectives showed that: (l) insti- 

tutes differed in the attention they gave to a variety of objectives; and, 

(2) participants, staff members, and directors disagreed in their evaluations 
of what happened at the institutes. Similar findings were obtained with 
regard to the preferences of the respondents. Thus, some institutes effec- 
tively met the interests of their participants, while others deviated con- 
siderably. 

A final examination sought to highlight the differences between highl- 
and low-rated institutes. While we noted t fiat diverse conditions could 
account for the rating of an institute program, a frequent finding was that 
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the high-rated institutes received almost consistently favorable comments, 
whereas the low-rated institutes had numerous negative remarks. In general, 
high-rated institutes seemed to provide rewarding s taf f/d i reccor relations 
and we 1 1 -organ i zed programs. Low-rated institutes, on the other hand, tended 
to be criticized, with comments generally directed to a variety of conditions. 

Unfortunately, preliminary reviews of proposals and catalogs (of the 
host colleges and universities) provided few clues to account for the differ- 
ences between high- and low-rated institutes. There were, however, a number 
of promising leads to warrant further research in this regard. 



PROJECT 2 



Specifically, the objectives of Project 2 involved the teacher- 
participants who attended NDEA summer institutes, and investigated: (1) 

subject matter content and methodology; (2) personal and professional atti- 
tudes; and, (3) teaching strategies. information for evaluating the success 
of summer institutes was obtained by measuring subject matter and methodol- 
ogical know-how through a simulated teaching process. This required simula- 
tion of a worksample of teaching behavior by developing a collection of items 
which were representative of the activities performed by teachers during a 
regular workday. Such activities were planning a unit or course outline, 
explaining a discipline's methodology and structure, discussing current ap- 
proaches of a subject matter, ordering educational aids necessary for class- 
room presentations, and answering inventories investigating professional 
affiliations. Nine situations of this nature, referred to as the "Teacher's 
Mailbox," were presented to each participant in the form of memoranda, letters, 
questionnaires, and requests for information. In addition, a specially con- 
structed Attitude and Opinion Survey and the A1 1 port-Vernon-L i ndzey Study of 
Values were administered to determine the degree to which summer institutes 
influenced the teacher-participants to change their value systems, self- 
concepts, aspirations, feelings toward children, notions of classroom disci- 
pline, and subject matter preferences. 

Scoring keys for the Teacher's Mailbox items were developed on two 
bases: a content analysis of the pretest returns, and the counsel provided 

by subject matter consultants. In this way, reality (content analysis) was 
combined with theory (expertise). Two scoring systems for each Mailbox item 
were also adopted: The Quantitative Score was obtained by counting the number 

of content categories given by a respondent, while the Qualitative Score was 
an overall rating of this answer. Every answer sheet was given at least two 
independent assessments by subject matter specialists. Scoring the Attitude 
and Opinion Survey and the Study of Values was performed according to pre- 
viously developed procedures. 

Since the Project 2 design called for pretesting and posttesting, teachers 
were contacted during the spring of 1967 and asked for their cooperation in 
the present study. Those who volunteered to participate were mailed the three 
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project Instruments in May, and three, again, in November of 1967. In this 
way, measures were gathered on these teachers both before and after their 
attendance at summer Institutes. The final sample, representing partici- 
pants from 9^ Institutes, consisted of 329 teachers, those who had responded 
to both administrations of the project Instruments, 

A repeated measures analysis of variance design was used and Interpre- 
tations of the results made by both the PROJECT IMPACT research staff and 
the subject matter consultants. The general conclusion drawn, for our 
sample of teachers, was that there were no signigicant changes In (l) Mail- 
box performance, (2) personal attitudes, (3) professional affiliations, 
(4) value patterns, or (5) teaching strategies. These results do not 
necessarily Indicate that the participants did not gain new Insights about 
subject matter content and teaching strategies, but rather Illustrates 
the difficulties so frequently encountered In seeking to show pre-post 
differences. The present approach contributed, however, a new dimension 
by permitting clinical assessments of the Institute program for future 
restructurl ng, The Teacher's Mailbox allowed us to consider Just such 
comparisons. Using: (l) the results of our analyses, (2) the written 

responses of the participants, and (3) the Insights of discipline specialists 
we were able to evaluate the summer institute program In terms of Its 
effects on each teacher-participant and the overall Impact of Individual 
Institutes for Improving the quality of teaching. Discipline experts 
were able to assess the styles and information used by the teachers In 
their simulated teaching procedures and Judgments made for correcting 
deficiencies, Furthermore, It was clear that teachers who attended 
summer Institutes continued to feel positively about their Institute 
experiences. That Is, objective measurement of institute Impact may have 
Indicated little, If any, change by participating teachers, but 
subjective evaluation of their expectations of Institute effectiveness 
continued to remain high, 



PROJECT 3 



The major focus of Project 3 was to determine the Impact of the NDEA 
summer Institutes on the colleges and universities which hosted them. As 
the third pilot study, this project elected to sample a small number of 
Institute directors and faculty members, seeking to learn of any Implemen- 
tation of Institute practices In their normal professional activities as 
a measure of impact. 

To Insure that we obtained answers to Important and relevant questions, 
we decided to use a structured Interview schedule. This procedure of asking 
all respondents the same questions maximized the number and comparability 
of responses while minimizing Interviewer bias and Intuitiveness. We con- 
tinued to Include the four subject mutter areas selected for Projects I and 2 



Two considerations guided our selection of Institutes: (1) university 

enrollment (we tried to choose three Institutes In each discipline, accord- 
ing to university size: small, medium, and large), and (2) the number of 

years that these universities had hosted Institutes, with two years required 
as a minimum. Our total sample, then, consisted of twelve Institutes, three 
per subject area. All Interviews were conducted Individually, with the 
Interviewer traveling to each of the twelve colleges or universities to talk 
with as many Institute directors, Institute staff members, and related 
^acuity members as possible. Difficulties of scheduling and meeting arrange- 
ments precluded obtaining the same number and types of Interviews at all 
twelve host Institutions. Interviews were tape recorded where possible, with 
approximately 90 percent of the Interviews so handled. All Interviewing was 
done during the fall semester following the 1 967 summer Institutes. 

Host of the taped Interviews were brief, averaging about thirty minutes. 
Much of the Information offered was of a descriptive nature In answer to the 
questions asked by our Interviewer, The scoring procedure was a content 
analysis, and the frequencies of responses were tablulated to each question 
across people, using a single scoring key for all four areas. No Individual 
w*s given a score of any kind, nor was any attempt made to weight the 
responses according to any predetermined definition of Impact. Two raters 
Independently scored each tape. 

While we must caution against any broad generalizations of the present 
flnd*ngs to all NDEA Institutes, based on twelve summer Institutes, the 
results do offer Interesting Indications of Institute effectiveness. Thus, 
respondents from all four subject matter areas gave generally favorable 
reactions to the Institute experience. Although some positive reports 
tended to be guarded, with some negative comments, the most common positive 
statement concerned the pleasant, rewarding nature of the Institutes. In 
addition, It seemed to have awakened many faculty members to their responsi- 
bility In Improving teacher education. Staff members from educational media 
Institutes reported the most overall change; political science showed the 
least change, although some departments did modify their regular academic 
programs In some way. 



A FINAL STATEMENT 



In conclusion, the general consensus of comments by directors, staff 
members, and participants describing the 1 967 NDEA summer Institute program 
was favorable, many stating that the Institutes had effected some changes 
In their consideration and education of teachers. While we were unable to 
show statistically significant pre- and post-institute differences with 
regard to Improved subject matter achievements, we can conclude that a 
majority of participants enthusiastically endorsed and recommended their 
Institutes, with a number stating that It was the best educational experi- 
ence they had ever had. In fact, we would argue that this Initially posi- 
tive approach by participants to the educational process Is the sine qua 
non for attitude and behavior change In the classroom. 
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INTRODUCTION 



The National Defense Education Act was passed by the Congress of the 
United States in 1958. This bill was developed to create a comprehensive 
plan for supplementing education in the United States. With the unrelenting 
increase in apparent knowledge within every discipline, the concerns for 
dissemination and utilization had become pressing. Summer training fcr 
elementary and secondary school teachers in their respective subject matter 
areas was provided by the Congress as one solution to this urgent national 
educational need* 

The reasoning and ideation behind the goal to improve the content and 
quality of American education need neither explanation nor justification. 

Few can disagree that education is our most important commodity and a primary 
guarantee for a better tomorrow. The question is, therefore, not with the 
"WHY" of such legislation, but rather with the "HOW." Few disagree that 
American education is in need of major repair. We need only note the many 
indicators that present practices are inadequately meeting the needs of 
students and teachers. 

While we grant the simplicity of this statement in describing current 
conditions of American education, it is not our purpose to discuss the nature 
of the system or to offer a blueprint for its revision. The subject of the 
present report is to consider the merits of the NDEA summer institute program 
That is, the primary objective of PROJECT IMPACT was to be the first inde- 
pendent evaluation of an endeavor to supplement the knowledge of classroom 
teachers by offering them a summer period of concentrated training in their 
respective subject matter specialties. 



AN ORIENTATION 



Before we describe our approach and findings, we should like to call 
attention to the following four issues, consideration of which, we believe, 
will contribute to a more meaningful and lucid interpretation of this evalu- 
ation. We recognize that for many this brief discussion may be unnecessary, 
but we suspect that it is preferable at this time to look on the side of 
commission rather than omission. 

Of first importance is the fact that no single technique of measurement 
can be expected to satisfactorily supply all the answers we usually ask of a 
research study. While strict adherence to exper imental design procedures 
can often minimize errors of all sorts, it is particularly difficult to 
follow such an approach in the educational domain where we find an intricate 
complex of subject matter, teaching methodologies, and individual differences 
It is important, therefore, to remember that a variety of uncontrolled vari- 
ables operate in a classroom, and any attempt to identify cause and effect 
relations between teaching and learning may from the beginning be idealistic 
and impract i cal . 
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Related to and following directly from the first issue is a second one 
which points out that the major goal of educational evaluation is the con- 
tinual assessment of a program's effectiveness — to determine its strengths 
and weaknesses* and to revise accordingly. It is inefficient and unrealistic 
to accept or reject an entire program on the basis of a single set of statis- 
tical findings. 

A third point we regard important is that evaluation can be "of the 
wrong kind, at the wrong time, and for the wrong reason." Careful planning 
helped PROJECT IMPACT heed these warnings: From the beginning, this study 

was: (1) not over-dependent on so-called "hard data" from indiscriminate 

use of standardized tests; (2) not undertaken too early in the development 
of the institute program; and (3) not concerned with final results in the 
statistical sense only. 

The fourth issue, while unlike the previous ones, is perhaps of greatest 
immediate and practical significance. Here, we refer to assaying the concept 
of any training program as an ideal process for improving the content and 
quality of American education. That is, the originators and developers of a 
program are likely to feel that theirs offers the solution to all difficul- 
ties. This kind of thinking, while understandable, is undoubtedly naive. 

On the other hand, it is equally naive to accept early, negative results as 
sufficient evidence for the abandonment of an entire program. 

In other words, no idea is at its inception likely to prove utopian. 

It is only after considerable effort is made toward clarifying crucial fac- 
tors, and then attempts are made to make indicated revisions that a project 
which appears initially promising does, in fact, fulfill its expectations. 
This is a roundabout way of saying that the NDEA summer institute teacher 
training program, while not without faults, was a relevant and practical 
approach. Arguments implying that content, the basic ingredient of NDEA 
institutes, is not basic to good teaching are superficial and downright 
foolish. At the same time, the need to stress affective education, ad- 
dressing teaching material toward the students' feelings as opposed to 
cognitive or intellectual learning, dare not be ignored. While PROJECT 
IMPACT was in its evaluation approach bound to the NDEA institute objectives, 
it hoped its basic orientat ion of continuing assessment would separate the 
wheat from the chaff. Above all, it sought to avoid labeling the summer 
institute program as either a "dramatic success" or a "dismal failure." 



GENERAL OBJECTIVES 



PROJECT IMPACT is best presented as three separate projects, each 
describing a distinctly different aspect of the impact of the NDEA summer 
institute program. Project l, involving teacher-participants, teaching 
staffs, and institute directors, asked their opinions relative to general 
institute performance and particular subject matter coverage. In Project 2 , 
we focused on the teacher-participants only, investigating the effects of 
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institute attendance on their subject matter learning of content and teaching 
strategies. Project 3 concentrated on colleges and universities which hosted 
these summer institutes, specifically seeking to explore some of the reac- 
tions and experiences affecting key educators at these educational institu- 
tions. To capture the full intent of the NOEA institute program, we found 
It necessary to develop, for Projects 2 and 3, sensitive and original measur- 
ing techniques. Accepting the difficulty and expense of such an undertaking, 
we agreed to operate on a pilot basis, selecting a small sample of respon- 
dents for inquiry and analysis in both projects. Accordingly, this approach 
limited final interpretations and generalizations. 



THE SUMMER INSTITUTE PROGRAM 



To the teaching profession of the not-too-d is t ant past, summers meant 
traveling, relaxation, a respite from books and pupils' dirty looks. For 
the present crop, however, summer is a time for retooling. Large numbers of 
teachers return to school each summer to take supplementary work, primarily 
to increase their subject matter knowledge, and, if for no other reason, to 
upgrade their academic standing. 

Recently, however, college and university campuses during the 
summer reflect a new vigor, an atmosphere of enthusiastic, spirited devotion 
to learning. It seems as if teachers, almost en masse , are returning to 
lecture halls, libraries, and laboratories. What makes it even more surpris- 
ing is that competition is keen and payment is received for this experience. 

It's all part of the National Science Foundation, the National Defense Edu- 
cation Act (especially Title Xl), the Higher Education Act of 1965, the Elemen- 
tary and Secondary Education Act, and finally, the Education Professions 
Development Act. 



Under the National Defense Education Act (NDEA) passed by the United 
States Congress in 1958, programs were established to improve the quality 
of teaching and to provide training for educational personnel in order to 
help meet critical shortages in numerous disciplines. To accomplish these 
goals, the now well-accepted summer institute plan which has provided 
unparalleled training opportunities for thousands of school teachers was 
organized. At the same time, hundreds of college professors of diverse 
disciplines were brought into the task of training school teachers, a’~task 
heretofore regarded as taboo* 

In fact, the hope was that as a result of this interaction, a continuing 
dialogue between the universities and the elementary, as well as secondary, 
schools would ensue. This interaction was to result in close cooperation 
between the trainers of teachers and the teachers themselves, thereby ulti- 
mately insuring commitment and dedication by both groups to teaching children. 
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What, then were the patterns of these summer institutes, to which so 
many teachers seek entry? The NDEA and Arts and Humanities Institutes 
were designed for twelve subject matter areas Including those areas 
of English, foreign languages, history, political science, geography, read- 
ing, teachers of the disadvantaged, and the creative arts. Typically, summer 
institutes enrolled about 40 elementary or secondary school teachers who were 
given daily instruction for six to eight weeks. Two or three courses were 
generally offered in addition to a "workshop" covering discussions of 
instructional methods and materials. The institutes often concentrated on a 
common subject matter theme, a set of general problems, or addressed them- 
selves to articulating some of the instructional strategies specific to a 
discipline. For the most part, the courses were organized to present new 
material to teachers who had already been well trained instead of concentrat- 
ing on remedial upgrading of poorly prepared teachers. 



THE SAMPLE 



Since there were 12 different areas covering 493 institutes involving 
some 20,000 participants, it was clear that decisions had to be made for 
studying only a few of these to represent the NDEA summer program of 1967* 

We were unanimous in our decision that only 4 areas were to constitute our 
sample. The areas selected were educational media (the "hardware" compo- 
nents), history (a previously studied area), political science ( the 
most recent institute additions), and instruction for teachers of disadvan- 
taged youth (the area for greatest potential attitude change). 

This plan resulted in a sample of 94 institutes, and consisted of 30 in 
history, 13 in political science, 24 in educational media, and 27 for teach- 
ers of the disadvantaged. Since approximately 30 to 60 participants were 
accepted by an institute, the total sample of teachers enrolled in the 94 
institutes came to more than 3 , 000 . With this relatively large number of 
subjects available, we felt confident that we would be able to draw upon a 
sample adequately representative for the development of educationally sound 
inferences vis-a-vis the institute program and its values to the teacher- 
part i c i pant. 




9 



18 



PROJECT 1 



In a sense, Project 1 is deservedly “No. I 11 because it was the most 
general in nature, reflecting the opinions and preferences of a majority of 
institute participants and staffs. Furthermore, the data gathered in this 
study permitted us to offer to the USOE its most important evaluation prod- 
uct — information for improvement and innovation. This study considered what 
actually took place at our sample of 9*+ institutes. Our plans centered on 
trying to relate stated summer institute objectives to actual activities 
offered at the institute. We wanted answers to a number of pertinent ques- 
tions. What material was covered? What procedures were followed in present- 
ing subject matter content? To name a few questions: Did an institute 

discuss such topics as professional journals and associations, classroom 
discipline measures, test construction techniques, audiovisual materials, 
and how children learn? Visits to each institute would have provided 
answers to these and many other questions, but this was not possible. 

Instead, we compromised by developing a questionnaire for obtaining the 
reactions of various members of the summer insti tutes--part ici pants , staff 
members, and directors. From our questions, we wished to learn how much 
emphasis each institute gave to a variety of teaching objectives. We recog- 
nized, of course, that respondent percept i ons would not mirror precisely the 
actual institute activities, but we expected that these would yield much 
usable informat i on. 



GENERAL PROCEDURE 



Our first move was to construct a measuring instrument, the Survey of 
Institute Objectives, to be administered several months after the completion 
of 1967 summer institutes. Each subject matter survey required that separate, 
but similar procedures be followed in its construction. This was necessary 
because each area had its own specific content and although institutes within 
the subject matter differed widely, they were much more homogeneous within 
the one area than among the four. The development of this scale involved: 

(1) perusal of pertinent summer institute proposals for specific and general 
objectives, (2) independent assignment of objectives into 1 1 1 i ke* 1 groups by 
four IMPACT staff members, and (3) pretes ting along wi th final reviews by 
measurement and subject matter specialists. 

The Surveys of Institute Objectives were mailed during the late months 
of 1967. Political science surveys were forwarded earliest, in November, 
and Surveys dealing with teaching the disadvantaged were distributed last, 
after Christmas. To insure maximum return of the Surveys, we asked the 
respective professional associations to assist us; after all, the information 
desired about each subject matter area was of considerable interest to those 
within the profess ion who are becoming involved in teacher training. Each of 
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the four associations provided a letter of introduction signed by its direc- 
tor. This' letter described the purposes of the questionnaire and was sent, 
along with a copy of the appropriate Survey of Institute Objectives and a 
return prepaid envelope, to all participants, staff members, and directors 
in our sample of Sk institutes. The number of individuals who received 
these materials was 3,656. 



The Survey of Institute Objectives 



Each of the four Surveys of Institute Objectives is a 50 to 60 item 
questionnaire dealing with the amount of attention placed on certain impor- 
tant objectives by a summer institute in that subject area. Alternatives 
indicating how much emphasis was given varied from "none 11 to "very much" 
and all respondents (teachers, staff members, and directors) were asked to 
check the alternative which most nearly described their reactions to each 
statement. The questionnaire statements were categorized by to[-ic, with 
7 to 10 groups in an instrument, and anywhere from 2 to 13 items within a 
category. There were obviously no right or wrong answers; only personal 
opinions and/or preferences were involved. 

A final part of each Survey asked respondents to evaluate personally 
the importance of each of the major categories. That is, regardless of the 
attention given to these major topics by their institutes, the respondents 
were asked how important they felt each category was to them. This informa- 
tion gave us further insight regarding what subject matter content the 
participants wanted covered by their institutes. Again, the data would com- 
plement related information we expected to get from the Project 2 measuring 
instruments. 

Scoring the Surveys consisted of coding each returned Survey with iden- 
tification numbers signifying respondent, institute, and status of respondent 
(that is, participant, staff member, or director). The answers to the items 
were scored by assigning one point for an answer of "none" to five points 
for an answer of "very much." Average scores were computed for both the 
individual objectives and the categories of objectives listed in Section I. 

In this way, we could compare institutes by objective and by category of 
similar objectives. Average scores were also computed for Section II of the 
Survey, providing respondents’ preferences for categories. This permitted 
us to compare the needs of the institute respondents with the actual activi- 
ties of the institutes. 



THE RESULTS 



The returns of 
for the four areas, 
questionnai res. 



the Surveys of Institute 
and were higher than are 



Objectives averaged 61 
usually found for most 



percent 
mai 1 
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These results are presented In Table 1, 



Table 1 

Survey of Institute Objectives Returns 



subject Matter Area 


Surveys Sent Out 
n % 


Surveys 

n 


Returned 

% 


Educational Media 


1080 


100 


752 


70 


H i story 


1050 


100 


709 


68 


Political Science 


539 


100 


276 


51 


Teachers of 
Disadvantaged Youth 


987 


100 


*»99 


51 


Total 


3656 


100 


2236 


61 



There was an Interesting agreement among the areas In their respective 
rates of return: educational media and history had comparable rates ( with 

70 percent and 68 percent, an average of 69 percent ), and political science 
and disadvantaged youth had Identical rates (51 percent). The net average 
difference between these two pairs of oddly matched disciplines seems large 
(18 percent), but whether It has any significance for our immediate purpose 
is not at present discernible. Perhaps, for the moment, it Is most reason- 
able to assume no explicable basis for this disparity in returns. 



Analysis of the Items 



Attention to Objectives 

Two conclusions are clear from our data, although neither one Is 
surprising. The first is that lnstl tutes di f fered it the attention they 
gave to a variety of objectives. The second Is that participants, 
staff members, and directors disagreed In their evaluations of institute 
performance. These two conclusions held across all four subject matter 
areas. 

The variations we found are shown In Table 2, which gives the lowest (0) 
and highest (5) participant values in each discipline (average value * 2.5). 
This same range held for staff member and di rector values. It is interesting 
to note that history Institutes had the widest discrepancy (2.0), while 
Institutes for teaching the di sadvantaged vart ed least (1.1). A possible 
explanation may be that differences are related to a change in objectives 
from the time the proposals are submitted to the time the Institutes are 
conducted. 
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Table 2 



Highest and Lowest Values Reflecting Attention Given 
to Institute Objectives by Participants 



Value 


Educational 

Media 


History 


Pol i t i cal 
Science 


Teachers of 
Disadvantaged 
Youth 


Highest 


A.o 


k.k 


OO 

CO 


A. 3 


Lowest 


2.6 


2. A 


2.6 


3.2 



Most perplexing was the disparity among those who were students and 
those who taught at or directed summer institutes. While we did not expect 
perfect consensus on the kinds of information presented at institutes, we 
did anticipate moderate agreement among the groups involved. The correlations 
presented in Table 3, however, indicate otherwise. 



Table 3 

Correlation Coefficients -Between Participants, Staff Members 
and Directors — Showing Their Attention to Categories of Objectives 



Teachers of 

Educational Political D i s advantaged 

Media History Science Youth 



Participants & Staff 


• 56 


• A3 


.57 


.17 


Participants & Directors 


• 35 


.03 


• 3A 


.21 


Staff & Directors 


.A 2 


• A3 


.59 


.06 



Of particular concern here is the lack of agreement among participants, 
staff members, and directors of disadvantaged youth institutes. We appre- 
ciate why these discrepancies existed during the early years of the institute 
program. At the same time, however, we believe that this is an important 
finding and has far-reaching implications for developing educational prac- 
tices for teaching the disadvantaged. 



Attention to Categories of Objectives 

Analysis of the Surveys in terms of their major categories produced a 
number of interesting findings. These results are presented in Table k. 
Perhaps of greatest importance are the two categories attended to MOST 
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(highest Survey values) by our institutes. Thus, institutes constituting 
our two content disciplines were, as they should have been, MOST attentive 
to increasing subject matter knowledge. We were also pleased to find addi 
tional evidence for an old recurring theme that the NDEA summer institutes 
motivated their participants, the first and, perhaps, most important phase 
preceding positive change and impact. 



Table 4 

Survey Categories Attended to MOST and LEAST by Institutes 



Area 


MOST Attended 
to Category 


LEAST Attended 
to Category 


Educat ional 
Med ia 


Motivating the Participants 


Relating Media to Reading, 
Communication, and Learning 




Developing Professional ism 


Evaluating the Effectiveness 
of Media Programs 


History 


Increasing Knowledge of 
Subject Matter 


Developing Professional ism 




Motivating the Participants 


Emphasizing the Learning 
Process 


Political 

Science 


Motivating the Participants 


Emphasizing the Learning 
Process 




Increasing Knowledge of 
Subject Matter 


Developing Professional ism 


Teachers of 
Di sadvantaged 
Youth 


To Identify General Aspects 
Relevant to Dealing with 
Disadvantaged Youth 


To Promote the Appropriate 
Use of Media and Materials 




To Develop Knowledge of 
Community Agencies That 
Aid Disadvantaged Youth 


To Develop Professionalism 
and Professional Skills 

To improve Teaching Ski 11s 


Also of 


interest are the LEAST (lowest 


Survey values) attended“to 



institute objectives. Granting that some categories must be sacrificed, the 
two categories so designated seem to be the proper ones. Developing profes- 



s i onal i sm 
knowledge 
very much 



i s not , we be! ieve , 
for emphasizing the 
a mystery. 



a first priority. Similarly, the necessary 
learning process still remains, we suspect, 



erJc 






23 



Preferences for Institute Objectives 



The second index value from the Survey of Institute Objectives sought 
to evaluate how important the major objectives were to participants, staff 
members, and institute directors. Our interest here was a comparison of 
the importance of objectives to respondents with the attention given to 
these objectives by the institutes. We derived this index by subtracting 
the "objective values" from the “preference values," hence the negative signs. 

Again, we found extreme variability among the institutes. The lowest 
and highest participant index values are given in Table 5* That is, some 
institutes effectively met the interests of their participants while others 
deviated considerably. Although we have presented values for participants 
only, the same discrepancies held for the staff members and institute direc- 
tors. This finding was, of course, unexpected in the case of directors, 
since they were responsible for the organization and development of their 
institute program from its inception. 



Table 5 

Highest and Lowest Index Values 
of Preferences of Objectives to Participants 



Index Val ue 


Educat ional 
Med i a 


History 


Political 

Science 


Teachers of 
Oi sadvantaged 
Youth 


Lowest 


-1.7 


-1.7 


-1.5 


-1.2 


Highest 


-0.3 


0.0 


-0. 1 


-0. 1 



Additional evidence for this divergence between interest of subject 
matter objectives and institute attention to these objectives is shown by 
the correlations presented in Table 6. Fortunate, however, are the high 
correlations between participants' stated interests and the attention given 
to these at all subject matter institutes. Apparently, the objectives con- 
sidered important to participants were those attended to at the higher-rated 
institutes, but overlooked at the lower-rated institutes. 
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Table 6 



Correlations Between Preferences of Objectives 
and Institute Attention to These Objectives: 
Participants, Staff Members, and Directors 



Group 


Educational 
Med i a 


Hi story 


Pol i t ical 
Science 


Teachers of 
Di sadvantaged 
Youth 


Part i ci pants 


cn 

CO 

• 


.85 


.89 


.87 


Staff 


.90 


.56 


.71 


.45 


Di rectors 


.56 


.16 


• 52 


.41 



Preferences by Category 

Our results describing the subject matter interests of directors, staff 
members, and participants provided additional insights about the institute 
program. Perhaps most striking was the small variability among major objec- 
tives of a subject matter area. Thus, it was clear from our data across all 
four disciplines that respondents, whether participants, staff members, or 
directors, considered most major categories of objectives listed in the 
Surveys about equally important to them personally. This was in contrast to 
their considerably greater variation in evaluating how much attention their 
institutes gave to objectives. In other words, it was frequently difficult 
to interpret the meaning of MOST and LEAST important because of the similarity 
in average ratings among the objectives listed in a Survey. There was no 
problem, however, in selecting the MOST and LEAST important categories. 

Table 7, which shows these results, lists (with two exceptions) only a single 
objective as MOST and LEAST important. Second place was often shared by as 
many as three other categories. 

We might also point out that our three groups of respondents were in 
less agreement with regard to the objectives which they preferred than 
they were in rating institute attentiveness. Thus, while Table 7 lists a 
MOST and LEAST for all three groups, this may represent a compromise where 
only two of three groups agreed. 
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Table 7 



Preferences by Category 



Area 


MOST Preferred 
Category 


LEAST Preferred 
Category 


Educational 
Med i a 


Developing S Administering 
“Media Service" and “In- 
Service" Program 


Relating Media to 
Read i ng , Commun i cat i on , 
& Learning 






Developing Competencies 
in Working with Media 


Hi story 


Increasing Knowledge of 
Subject Matter 


Emphasizing the 
Learning Process 






Developing Curricula 


Pol i t i cal 
Science 


Increasing Knowledge of 
Subject Matter 


Developing Profess iona 1 i i 


Teachers of 

Disadvantaged 

Youth 


To Increase Understanding 
of the Needs of the 
Disadvantaged Youth in 
the Classroom 


To Increase Knowledge 
of Curriculum & 
Curriculum Development 


The High and Low Institutes 



This part of the report describes our analysis of the relatively high 
and low institutes within each subject area of our sample. We hoped to 
learn the kind of factors responsible for these evaluations. What did par- 
ticipants look for in the NDEA summer institute program? What did they 
expect to get from the institute staff and director? How did they react to 
the teaching practices offered? Our desire to answer questions such as these 
and others provides, we believe, fundamental and insightful information to 
the U. S. Office of Education for: (1) revising summer institute programs , 

and (2) generalizing to other teaching and learning pract i ces--al 1 aimed 
toward increasing the effectiveness, relevancy, and efficiency of American 
educat ion. 

As a first step, we had to select our high and low institutes. After 
consideration of a large number of possible scores based on participant, 
staff, and director judgments, we limited ourselves to two evaluations. 

These were the values assigned by the participants to: (1) the objectives 

attended to by their institutes, and (2) the discrepancies between these 
ratings and corresponding interests in them. These two scores will be 
referred to, from this point on, as the Objective Score and the Interest 
Score , respect i ve 1 y . 
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Using these two criteria, we ranked all institutes in a discipline and 
then selected three to five of the highest and lowest institutes in each 
subject area. Table 8 presents the averages for the high and low groups. 
Inspection of these values shows identical patterns across the four disci- 
plines, with clearcut differences between the high and low institutes for 
each subject matter. 



Table 8 

Average Objective and Interest Scores 
for High and Low Institute Groups 



Scores 


Ed. Media 
Low High 


History 
Low High 


Pol i t ical 
Science 
Low High 


Teachers of 
Disadvantaged 
Youth 
Low High 


Object i ves 


2.9 3.9 


2.8 4.2 


2.8 3.7 


3.2 4.2 


1 nterests 


-1.3 -0.5 


-1.1 -0.1 


-1.1 -0.3 


-1.1 -0.3 



Information to help account for these differences comes from three 
sources: (l) comments on the Survey of Institute Objectives, (2) statements 

of purposes and procedure given in the original proposals, and (3) formal 
catalog descriptions of the colleges and universities hosting the institutes. 
We shall discuss each briefly, but the comments provided by participants 
offered the major source of data. 

To use these comments, however, we first had to organize them into 
meaningful and relevant groups. We developed a scoring key of some twenty 
such classes, including, for example, comments on the Staff and Director, 
Teaching Methods Stressed, Institute Organization, and Emphasis on Subject 
Matter Content. The same scoring sheet was used for the four disciplines 
and scoring required a tally of all comments as either favorable or unfavor- 
able. We shall present these analyses, by subject matter area, in the 
sections which follow. 



i 



Political Science 




Comments for the high-rated institutes were almost unanimously favor- 
able, many stating that the institute covered both content and teaching 
methods thoroughly. The following three groups receiving favorable ratings 
are consistent across the high institutes: (1) General Comments Describing 

the Rewarding Nature of the Institute Experience, (2) Reports of Positive 
Interactions Between Staff and Participants, and (3) Perceptions of In- 
creased Subject Matter Competency. 
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With regard to General Comments, participants enthusiastically endorsed 
and recommended their institutes, and a number stated that It was the best 
educational experience they had ever had. 

Comments on staff and Director indicated that, for the most part, staff- 
participant interaction was excellent. "We were treated as equals, and the 
seemingly great gap between the professor and student was bridged." Similar- 
ly, "The director recognized our need as secondary school teachers, not as 
political scientists, and did his best to meet them." 

Most of the comments speaking to either of the groups, Emphasis on 
Subject Matter or Teaching Methods included a comment about the other. For 
example, "The institute was excellent in its aim to give adequate opportunity 
for teachers to improve their grasp of content and methods in the political 
sciences." Also, "The comb i na t ion of civics and education gave me the basis 
to help my students." Other participants had somewhat different perceptions 
about the balance of content and method, but they were still pleased with 
what they received. One said, "I feel the institute attempted to make me a 
better teacher rather than giving me more information in political science." 

Low-rated institutes received more unfavorable comments than the high- 
rated institutes in each case. For the most part, these negative comments 
stressed the reverse of what appeared to happen at high-rated institutes, 
with additional negative references to poor institute organization. 

In general, most comments about the low-rated institutes were critical. 
There were complaints about planning and organization, the discussion session, 
and the theoretical nature of the subject matter with little attempt toward 
practical classroom application. The general consensus with regard to the 
three lowest-rated institutes was that "they were a complete disappointment 
educationally," although very few said their "institute experience was a 
failure or worthless." 

While it was difficult to tease out any clear-cut differences between 
the proposals of high- and low-rated institutes, it did appear that the high 
institutes were more general in their concerns for the content of political 
science. They did, however, speak pointedly to the importance of improving 
teaching skills and techniques. The low institutes specified highly detailed 
areas of content, sometimes esoteric in nature and difficult to insure under- 
standing by those in attendance. These institute proposals also appeared to 
minimize emphasis on teaching methodology. 

Another difference we noted was in the Special Criteria for Eligibility 
described in the original proposals. Th*. high-rated institutes specified 
few special criteria, and where they did list any, these were by geographical 
area or teaching grade level. The low-rated institutes, on the other hand, 
enumerated such criteria as general excellence in undergraduate studies and 
letters of recommendation from former professors and high school adminis- 
trators. 

Unfortunately, only superficial accounts were available from catalogs 
describing the institutions hosting political science institutes. Further- 
more, it was difficult to obtain comparable information from these catalogs. 
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One difference can be cited here, however. The three high-rated institutes 
were all conducted at universities with large undergraduate enrollments of 
over 20,000 students, while the three low-rated institutes were held at 
smaller institutions of less than 10,000 students. 



H i story 

Analysis of comments for the three Institutes considered in the high 
group showed that two of these received no unfavorable comments, while the 
third had a few criticisms. General Comments was the only category which 
received numerous positive comments for the three high institutes. Organi- 
zation of the Institute and Motivation Toward Professionalism were rated 
high In two of the three. 

The high institutes were described by participants as stimulating, 
enriching, effective, and beneficial. Thus, manyiquoting almost exactly the 
words of one participant said, "It was one of the best all-around learning 
experiences I have ever had." 

For the most part, the high-rated institutes were considered well- 
organized and well-planned, with the result that participants believed the 
institute had a major impact on their teaching by encouraging them to incor- 
porate many of the ideas and techniques of the institute into their courses. 

It was also clear from comments on Professionalism that the institute experi- 
ence motivated participants to keep up and to develop a spirit of pride in 
their field of hi story. 

The low-rated institutes are distinguished from the high group by the 
large number of negative comments, with no category rated favorably across 
all four low institutes. Three of these institutes received about equal 
positive and negative comments, and the fourth institute had almost all 
negative comments. No single category, however, was consistently unfavorable 
across all institutes. The Staff /Di rector category was rated low In two 
institutes, and the handling of Teaching Methods considered relatively poor 
i n two i nst i tutes. 

Responses in the latter category (Teaching Methods) further confirmed 
the observation thatH nsufficient emphasis was placed on teach i ng methods by 
the staffs of the four institutes rated as low. Respondents complained that 
the lecture method was used excessively, and that there was not sufficient 
class participation or discussion. Some participants also pointed out that 
the institute program "tried to do too much for the time allotted." 

In summary, we can say that the high institutes received almost consis- 
tently favorable comments, whereas the low institutes had numerous negative 
comments. Apparently, participants held polarized opinions of low institutes. 
Some Insight into reasons for these differences of opinion may be found in 
the comment of one participant who said, "...Several of the participants 
reacted so negatively to the political views of many staff members that objec- 
tivity of these survey responses may be seriously questioned." This dissonant 
attitude by participants toward staff political views, again reflected by 
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the comment, "I was extremely disappointed because the director tried to 
Impose his bias," has interesting implications for the acceptance of instruc- 
tion. It Is probably reasonable to assume that similar phenomena existed at 
several Institutes, although we have a direct suggestion of such a relation- 
ship from only one institute. 

A review of catalog descriptions for the seven Institutions 
included among the high and low history institutes afforded no clues regard- 
ing possible factors responsible for the institute evaluation ratings. Thus, 
geographical locations of north and south described universities hosting both 
high-rated and low-rated institutes. Similarly, university enrollments did 
not differentiate the highs from the lows, since all host institutions were 
schools with less than 10,000 students. It is also of interest to note that 
the seven universities represented were predominantly privately endowed, with 
one Catholic university in both the high- and low-rated groups. 

Examination of the proposals for the high- and low-rated institutes 
offers several interesting hypotheses which, after further study, could 
account for differences. For example, we can cite that the high-rated insti- 
tutes tended to spell out their content objectives more specifically. They 
also, at least in this situation, seemed to stress problems of twentieth- 
century American history in contrast to stated objectives of the low insti- 
tutes which preferred to relate American history to the larger domain of the 
past. 



While we could not detect differences in other factors such as Criteria 
for Eligibility of staffs selected to conduct the institutes, we do believe 
that the programs described for the high- and low-rated institutes were dif- 
ferent. Thus, we detected greater formality of teaching, in the usual graduate 
school manner at the lowest institutes. On the other hand, the three highest 
were conducted In a more permissive atmosphere, offering informal courses 
and seminars which were designed to meet the specific needs of the attending 
participants. 



Educational Media 

Participants attending educational media institutes were clear in what 
they expe“cted"f rom their NDEA summer institute experience. Thus, institutes 
in the high group were considered highly positive, with four among the five 
receiving almost all favorable comments. Two categories, General Comments 
and Staff/Director, consistently received positive comments across all five 
high institutes. Many participants offered statements similar to the follow- 
ing: (1) " 1 was tremendously pleased with the institute and believe that I 

obtained a wealth of knowledge on the use of different media 11 ; (2) "The insti- 
tute was a great learning experience"; and (3) "This summer was my most 
rewarding college experience." 

Staff/Director comments were also numerous and strongly positive. These 
reactions were described as competent, friendly and inspiring, always willing 
to go that "extra mile." A third category which received a large number of 
favorable reactions by participants of the five high-rated institutes was the 
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Carry-over of Institute Experiences to Home Classrooms. Here, the general 
consensus is well reflected by the participant who said, 1 1 1 am using more 
audio-visual materials than ever before, and hopefully, more beneficially. 11 
Related to these positive feelings of participants is the fact that whatever 
negative comments were offered, they were directed at excessive sitting and 
listening, with little participation in problem-solving. 

The low-rated institutes again received a majority of negative comments. 
That is, participants' statements would indicate that while the low-rated 
institutes had both good and poor components, the high-rated institutes had 
few faults. In fact, even when favorable comments were made about the low- 
rated institutes, they were, it seemed to us, generally less enthusiastic 
and persuasive than those describing the high group. Only a single category, 
Organization of the Institute, appeared to get consistently unfavorable 
reactions. Participants complained that the low-rated institutes were poorly 
organized, with attempts to do too much and to cover too many topics. 

A study of the nine applicable Institute proposals revealed 
three possible differences between the high and low media institutes. The 
first concerned the Criteria for Participant Eligibility. Here we noted 
that the high institutes were much more clear-cut and specific, seeking to 
demand that accepted participants show proof that they wi 1 1 be directly in- 
volved and responsible for a media program upon their return to school in 
the fall. In contrast, the low institutes stressed admission tests and 
grades. The second difference we noted was in the brief program outlines 
given in the proposals. It seemed to us that the high-rated institutes were 
specifically organized to survey the development and potential of educational 
media for today's classrooms while the low institute programs reflected a 
different orientation. Thus, one low-rated institute tooled up for instruc- 
tional TV practices only while another discussed process rather than specific 
content. A third possible contrast showed up in the staffing category where 
the high-rated institutes tended to select their staff members (with insti- 
tute experience) from their own university faculties, whereas a higher per- 
centage of visiting faculty members (without indications of institute back- 
ground) staffed the low-rated institutes. 

A review of college and university catalogs provided no clues 
for discriminating between the high and low groups of media institutes. 



Teachers of Disadvantaged Youth 

Analysis of the Survey comments from participants who attended 
institutes for Teachers of Disadvantaged Youth indicated, that 
these inst i tutes were considered either very high or very low. That is, 
participants from the high group offered no criticisms about their institutes, 
but respondents from the low group made numerous negative comments along with 
some positive statements. Furthermore, there were very few comments avail- 
able for any of the high institutes. As a result, it is impossible to assess 
any specific areas of emphasis for the high group. 
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In contrast, teachers from the low group were freer with their 
positive and negative evaluations, with criticisms accounting for approxi- 
mately one-quarter of all comments. In view, however, of the dearth of 
comments describing high institutes, we were at a loss to identify any con- 
sistent trends in the subject matter area. In fact, our general consensus 
was that the participants tended to contradict each other in their evalu- 
ations, making interpretations difficult, if not impossible. 

Strangely, the low institutes appeared to be rated favorably: (l) on 

developing insight in the background and make-up of d i sadvant aged youngsters , 
and (2) on improving the ability to work with and teach the disadvantaged 
more effectively. Remembering, however, that these institutes were estab- 
lished for teachers of the disadvantaged, any information directed to under- 
standing these young people may be judged to be valuable and positively 
reinforcing. The one category judged unfavorably by a number of participants 
at two of the low institutes was Organization of the Institute. In general, 
institute programs were considered unstructured to the point that some of 
the participants were not always sure what was taking place and others stated 
that there was much time wasted on inconsequential activities. While 
most participants complained about institute organization, there were some 
positive reactions, but these tended to be more general than the negative 
comments. 

A review of the proposals of the high and low institutes for teachers 
of the disadvantaged revealed three categories we judged to differentiate 
between the two groups. These were Criteria for Eligibility, Program Develop- 
ment, and the Staff /Di rector . Thus, the high institutes specified partici- 
pants who came as teams, particularly to include supervisors and/or principals. 
The low institutes sought classroom teachers with no additional specifications 
required. With regard to program development, the high institutes seemed to 
offer carefully thought-out programs. They succinctly clarified the roles of 
theory and practicums, with detailed descriptions of how they planned to work 
with disadvantaged children. Staff selection at the high and low 
institutes showed similar differences. Here, we found that the high insti- 
tutes emphasized the psychological and social backgrounds of their faculty, 
while low institutes preferred the educator, part i cu 1 a r 1 y wi t h university 
affiliation. It was also interesting to note that both high and low insti- 
tutes appointed staff members with previous institute experiences. 
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PROJECT 2 



OBJECTIVES 



Concerned only with the impact of institute participation on the teachers 
who attended summer institutes, the objectives of Project 2 concentrated on: 

(1) subject matter content and methodology, (2) personal and professional 
attitudes, and (3) teaching strategics. These were only three from among 
many factors originally considered, and included the following formal state- 
ments of their meaning and significance: 

(1) Subject Matter Content and Methodology - Skills of the discipline, 
understanding the structure of the discipline, participants' under- 
standing of the nature of their discipline, and sensitivity to 
discipline methodology and content. 

(2) Personal and Professional Attitudes - Commitment to the discipline, 
concern for teacher education, felt needs, values, self-concepts, 
and attitudes toward subject areas. 

(3) Teaching Strategies ~ Testing, use of pedagogical techniques as 
an integral part of the learning process, concern for awareness 
and use of learning principles, and individual differences. 



THE MEASURING INSTRUMENTS 



Content 



The realization that we were dealing with as many as 9^ possible tests 
meant, of course, that the number of objective test items required to measure 
these areas would be astronomical, not only in actual number but in cost as 
well. Furthermore, the emphasis of Title XI of the National Defense Educa- 
tion Act in funding summer institutes was to meet such broad goals as under- 
standing the nature and structure of a discipline, showing greater sensitivity 
to discipline methodology and content, having a commitment to the discipline, 
revealing a concern for teacher education, using pedagogical techniques as an 
integral part of the learning process, and showing interest in learning theory 
and individual differences. 

To measure these groups, we adopted a simulation approach. This method 
required us to simulate a worksample of teaching behavior by developing 5 
collection of items which were representative of the kinds of activities 
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performed by teachers during a regular workday. This approach is generally 
referred to as the "in-basket" technique and was first described by Norman 
Frederiksen for use with administrators. Teachers, however, receive almost 
all communications, memos, assignments, etc., in a mailbox rather than an 
in-basket. Hence, we designated our in-basket test the "Teacher's Mailbox." 



The Teacher's Mailbox 



The Teacher's Mailbox is a self-administered situational test in written 
format. It projects a teacher into a realistic situation, requiring him to 
handle tasks like those he encounters in his regular teaching activities. 

That is, the teacher-participant is asked to play a ro1e--that of a teacher. 
Through this kind of real-life involvement, we hoped to learn how tne teacher- 
participant had been affected by his institute experience — in respect to the 
substantive content and the methodology of his discipline and to the means 
by which he could communicate that knowledge to his class. This was, indeed, 
the closest we could come to observing the actual teaching behavior of the 
participant. 

Directions stressed that each participant play himself, in his own 
school, carrying out his regular teaching assignments. The only changes from 
the real-life situation were those involving the labels of individuals and 
places. Thus, we gave the examinee a new name, Richard Williams, whose school 
was Lindenwood School located in Linden City. His principal was named Robert 
Grant, and the superintendent of schools, Everett Collins. Other friends and 
colleagues, institutions and organizations, also bore fictitious names. These 
names were the same for all participants, but they represented a variety of 
different individuals and schools. As a final aid, we included appropriate 
supplies (stationery, envelopes, notepaper, pencils, pens, and paper clips) 
necessary for completing the Mailbox items. 

The situations which we developed for an area included items seeking to 
measure a participant's knowledge of his subject's: (l) technical vocabulary, 

(2) fundamental literature, (3) characteristic methodologies, (4) relation- 
ship to other areas, and (5) professional organizations. These items were 
then couched in terms of authentic on-the-job teaching assignments and teach- 
er chores, like memos, letters, and questionnaires. 

Very early in our instrument development, we believed that the same 
simulated situations might be used for all four disciplines, but this belief 
was found to be naive when we began in-depth analysis of subject matter con- 
cent. As a result, we scrupulously avoided forcing similarity across subject 
matters, although we were able to adapt one or two topics for all areas. 

After careful review, pretesting, editing, and revision, we were satisfied 
that the nine items selected in each subject matter area adequately sampled 
the appropriate content and met our objectives for Project 2. 

Scor i ng the Mai 1 box . Scoring keys for the Teacher's Mailbox items were 
developed on two bases; a content analysis of the pretest returns, and the 
counsel provided by subject matter consultants. This approach was our way 
of combining reality (content analysis) with theory (expertise). After they 
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were independently reviewed by subject matter specialists, the subclasses 
derived from the pretest content analysis by PROJECT IMPACT staff members 
and the "ideal 11 answers supplied by subject matter specialists were incor- 
porated into a single scoring key for each Mailbox item. Each scoring key 
was arranged to include a listing of subclasses on the right-hand side of 
the page, while the left was reserved for indicating the presence (stated) 
or absence (not mentioned) of the topic in a participant's answer. 

To learn as much as possible about a participant's knowledge of his 
subject matter, we decided to use two scoring systems for each Mailbox item. 
These we called Quantitative Scoring and Qualitative Scoring. The Quanti- 
tative Score was obtained by simply counting the categories checked in the 
Stated Column. 

The Qualitative Score was selected to give an overall evaluation of 
the respondent's answer, regardless of what contributed to his Quantitative 
Score. The Qualitative Score was a rating based on a six-point scale rang- 
ing from unsatisfactory to excellent. in addition to the scale, every 
Qualitative Answer Sheet, one to each Mailbox item, briefly summarized for 
the scorer the major points that participants should include in their answers 
to that communication. And finally, examples of nine responses were provided 
to assist a scorer in establishing references for evaluating the response to 
be scored. The nine examples, three for each group, represented three quality 
ratings, Excellent, Average, and Poor. 

Every answer sheet was given at least two independent assessments. In 
all cases, the. subject matter scores included those individuals who had been 
asked to develop the scoring keys. 

Test Re 1 i abi 1 i ty * Recognizing the complexity of scoring, we agreed to 
the importance of having two scorers serve as » .-.dependent checks on one 
another. This procedure also enabled us: (l) to estimate the reliability 

or consistency of the Teacher's Mailbox scoring procedures, and (2) to com- 
bine the two ratings for a more dependable final score. 

The reliability estimates were obtained by correlating the scores of 
our two raters for all Mailbox items and then getting an average correlation 
coefficient for the seven letters in each subject matter area. Table 9 
presents these results for both the Quantitative and Qualitative Scores. 
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Table 9 



Teacher's Mailbox Average Inter-scorer 
Reliability Correlation Coefficients 



Subject Matter 


Quant i tat i ve Score 


Qual i tat i ve Score 


Educational Media 


.65 


.85 


Hi story 


.62 


• 76 


Political Science 


• 56 


.58 


Teachers of 
Disadvantaged Youth 


• 54 


.56 



Unexpected 1 y, i n view of the few categories required for qualitative 
scoring, the reliability correlations are higher for this system than for 
the Quantitative Scores. The inter-scorer reliability differences between 
the two scoring systems are greater for educational media and history, but 
show little variation for either political science Mailbox items or those 
from the Mailbox for teachers of the disadvantaged. Similarly, we were not 
surprised to find that the rating system followed to score educational media 
responses was the most reliable, while scoring responses concerned with 
teaching the disadvantaged was least consistent. It was evident at the time 
the Mailbox letters were constructed that our educational media consultants 
were developing items highly specific to their area, whereas it was far more 
difficult even to determine a specific body of information, no less to write 
questions on it, that could be referred to as appropriate to working with 
disadvantaged youngsters. The diversity of items included in the Teacher's 
Mailbox for teachers of the disadvantaged attests to this difference, with 
scoring also being more subjective and relying more heavily on the personal 
interpretation and biases of the scorers. Measurement and scoring in the 
fields of history and political science would seem logically to fall some- 
where between these two extremes. Our data support th i s pos i t i on. 

Test Val id i ty . This discussion on the validity of the Teacher's Mailbox 
is based on data collected during the pretesting stage of the Ma i 1 box admi n- 
istrations. It presupposes knowledge of these pretest results and may not 
be completely clear until the results section has been read. We believe, 
however, that this preliminary analysis of the Mailbox validity is most ap- 
propriately presented at this point. 

Initial evidence of the validity of the Teacher's Mailbox for each 
subject matter area was obtained from comparisons of the performance of 
participants on five relevant dimensions. Thus, we compared the pretest 
averages of teachers from large and small schools, from elementary and higher 
public schools, from large and small communities, with B.A.'s and M.A.'s, 
and with more and less than seven years' teaching experience. Since similar 
results were obtained for these comparisons in all four subject matter areas, 
we shall illustrate with examples from a single discipline, history. These 
Mailbox data are presented in Table 10. 
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Table 10 



Teacher's Mailbox Pretest Averages 
History Participants 



Comparison Group 


Quantitative Score 


Qualitative Score 


Fewer than 900 Students 


23.1 


3.2 


More than 900 Students 


27.0 


3.6 


Cities with Fewer than 50,000 


23.3 


3.2 


Cities with More than 50,000 


27.6 


3.7 


Less than 7 yrs. of Teaching 


2 * 4.0 


3.2 


More than 7 yrs. of Teaching 


26. 1 


3.5 


Elementary Teachers 


22.3 


3.1 


Junior/Senior High Teachers 


25.8 


3.5 


B.A. Degrees 


2*4.5 


3.3 


M.A. Degrees 


25.8 


3.5 



Examination of the five groups of pretest comparisons shows the follow- 
ing: (1) the large school and large city teachers outscored their small 

school and small community co-workers; (2) participants with seven or more 
years of teaching experience and/or teaching in junior-senior high schools 
had higher pretest Mailbox averages than those who had less than seven years 
of teaching experience and/or were elementary school teachers; and (3) 
teachers with Master* s degrees tended to perform higher than those with only 
Bachelor's degrees. The superior performance of the experienced and better 
educated teachers, while not necessarily statistically significant, is in 
the desired direction and provides favorable evidence about the validity of 
the Mailbox letters. Further research and analysis are, of course, neces- 
sary before we may conclude that the Teacher's Mailbox is a valid assessment 
i nst rument . 



Attitude Assessment 



The assessment of attitudes in behavioral evaluation studies has long 
been popular. Since society today requires a far greater understanding of 
attitudes, their measurement is an indispensable part of behavioral investi- 
gations. Hence, PROJECT IMPACT recognized the importance of attitudes, their 
formation as well as their covert and overt manifestations, to the teaching 
profession. 
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A fundamental and apparent dimension of impact is the change in teacher 



attitudes resulting from the institute experience, although we readi 
that a six-to-eight week period is a very short span of time for maj 
enduring attitudinal changes. Attitudes reflect deep-seated disposi 



ly admit 
or or 
t ions 



which are not easily modified. Furthermore, the primary institute objective 
almost inevitably concentrated on subject matter content, barely involving 
attitudinal factors, at least not deliberately or consciously. Since, how- 
ever, a favorable attitude is probably at least a necessary, though not 
necessarily a sufficient reason for mastery of new subject matter materials, 
we felt compelled to learn something about teachers' values, self-concepts, 
aspirations, feelings toward children, notions of discipline, and subject 
matter preferences. We, therefore, developed the PROJECT IMPACT Teacher 
Attitude and Opinion Survey. All items were of the objective variety and 
the Survey was completely self-administering. 



fol lowing 
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The Attitude and Opinion Survey 

We drew the majority of the items for this instrument from the 
sources. The "Attitude and Gratification Preference Schedule" and 
vation for Teaching" both by Stern and Masling furnished the bulk of 
material for Scales i and II. Scale III was based on the "Teacher 
istics Schedule," an instrument resulting from Ryans' comprehensive 
taking research assessing teacher characteristics. The last section 
IV, was taken from an inventory developed by Prince to answer what 
the differences in value structure, specifically, "traditional" vs. "emer* 
gent," may have on the satisfaction, confidence, and effectiveness of indi- 
viduals within the school system. 

The 20A items comprising the PROJECT IMPACT Attitude and Opinion Survey 
were scored by adapting the scoring systems of the original scales, from 
which these items were taken, to the needs of Project 2. We felt it was 
necessary to modify the scoring keys because the three original attitude 
scales had been developed to meet objectives somewhat different from ours. 

Our sample for this part of Project 2 included all participants who had 
completed both the pre- and post-Attitude and Opinion Surveys, regardless of 
whether they had returned the Teacher's Mailbox items. Using this group as 
our sample increased our total number to 688 and permitted several statis- 
tical analyses requiring a larger number of cases than our pre-post [sample. 

Related to the concept of attitude is another of our Project 2 igoals, 
that is, to consider the role of value characteristics in the teaching 
process. For our study, a necessary corollary was to determine what, if any, 
changes in the value structure of teachers we might expect from a summer 
institute experience. Again, we were probing for additional personality 
information because the personality of the teacher is so basic to teaching 
performance. 
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The Study of Values 



Since we viewed Project 2 as a pilot study, we preferred to select only 
a few relevant factors characterizing the dynamics of teacher personality, 
i.e., his attitudes, values, and self-concept. In addition, we wanted a 
we 1 1 - researched instrument to serve as a kind of bench mark against which we 
could compare the results from our newly devised Teacher*s Mailbox and Atti- 
tude and Opinion Survey. Our choice was, then, the A1 lport-Vernon-Lindzey 
(AVL) Study of Values. This Study of Val ues is not only an objective, self- 
administering instrument, but it is a^so easily scored and interpreted by a 
respondent. We encouraged our participants to score their own scales, hoping 
that the results would prove personally informative and rewarding. Further- 
more, we believed that this scoring would reinforce our assertion that the 
instruments we developed were not meant to be tricky, misleading, or decep- 
tive. They were one part of a sincere and honest attempt to appraise the 
impact of the summer institute program. 

The AVL Study of Values seeks to measure the relative prominence of six 
basic personality interests or motives: theoretical (interest in the dis- 

covery of truth), economic (interest in what is useful), aesthetic (emphasiz- 
ing form and harmony), social (valuing altruistic love or philanthropy), 
political (interest in power, although not necessarily in politics as such), 
and religious (seeking of unity, relating the self to the totality of the 
cosmos) . 

The AVL is an ipsative scale. That is, a high score on one of the six 
scales can be achieved only by obtaining a lower score on at least one of 
the other five scales. Therefore, the Study of Values indicates the relative 
importance of these six values for an individual, rather than their absolute 
importance. The fact that one respondent is highly concerned with the values 
while another feels quite d i fferent ly wi 1 1 not be reflected in their scores. 
Rather, only the relative importance of the values will be provided. 

The scale consists of two parts requiring a total testing time of about 
15 minutes. Part I lists 30 controversial statements or questions and asks 
the respondent to indicate his personal views by selecting the alternative 
that is relatively most acceptable. Part II contains 15 situations or ques- 
tions followed by 4 possible attitudes or answers. The respondent is asked 
to arrange these answers in order of his personal preference. 

Previous research with the AVL has shown that significant differences 
in values do exist among teachers in different subject matter areas. We 
questioned, however, whether the relatively short summer institute program 
could seriously affect the resulting va 1 ue pat terns. 
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WHAT WAS DONE 



Selection of Subjects 



Recognizing the pilot or exploratory nature of the study, we agreed that 
a mcdest though adequate sample size would be 100 teachers in each subject 
matter area. We then sought to determine how many participants would have 
to be contacted at each stage of our study in order to reasonably expect 100 
usable subjects in each area. 

Five principles guided our thinking at this time. Foremost, we acknowl- 
edged that our teachers as research subjects could not be expected to coop- 
erate any more than other research subjects, notwithstanding the fact that 
the teachers had been chosen to attend summer institutes. Second, Project 2 
was to demand much more from a subject than does the typical evaluation study 
Next, participation was on a volunteer basis. Fourth, the timing for pre- 
testing was an unfortunate selection, coming during those inevitably busy and 
exhausting final weeks of the school year. And finally, we recognized the 
pitfalls of having to adopt a mail approach due to the problem and costs 
associated with an initial plan of establishing testing centers strategically 
dispersed across the country. 

These constraints helped us develop a realistic forecast as to the 
attrition we might encounter at the various states of our data collection. 

If we started with a group of 1,000 participants in subject matter and antici 
pated that 50 percent of the group would volunteer, we should get a sample of 
500 teachers of pretesting. Then, expecting 50 percent of this spring pre- 
testing sample of 500 teachers to participate, we assumed 250 would return 
the test materials. And finally, our judgment, plus a study of results ob- 
tained in survey research in general, suggested only a 40 percent return at 
fall posttesting when test kits would be forwarded to those 250 who had re- 
turned pretests. The 40 percent return at this time would provide the final 
pre-post samples of 100 teachers in each discipline. 

The returns by subject matter area are shown in Table 11. You will 
note, of course, that our actual total volunteer sample, 48 percent, closely 
approximated our prediction of 50 percent, with the smallest percent of 
history teachers volunteering (43 percent), and educational media personnel 
being most agreeable (54 percent). 
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Table 11 



Teacher-Participants Who Volunteered for Project 2 



Subject Matter 


Letters 

n 


Sent 

% 


"Yes" Postcard 
n 


Retui 

% 


Educational Media 


841 


100 


451 


54 


Hi story 


1015 


100 


440 


43 


Pol i t ical Science 


972 


100 


311 


52 


Teachers of 
Disadvantaged Youth 


594 


100 


433 


45 


Total 


3422 


100 


1635 


48 



Predicted - 50% Returns 
Actual - Returns 



The representativeness of the sample is a consideration of importance 
in all research. Frequently, however, this may be only academic, since 
parameters are unavailable for determining the degree of representativeness 
attained. Fortunately, we were able to investigate some dimensions of our 
sample by comparing it on a number of biographical items with the total group 
of participants who attended our 196? sample institutes. This demographic 
information was obtained from the USOE Application Record Card. Some factors 
we believed might be pertinent and informative were the following: (1) age, 

(2) sex, (3) total years of teaching, (4) highest degree held, (5) under- 
graduate and graduate majors, (6) private or public school system, (7) school 
environment, (8) attendance at a previous NDEA institute, (9) level of school 
system, and (10) location of school. We are the first to admit, of course, 
that the fact that these factors were available and measurable does not 
necessarily mean that they are those meriting attention. Surely, however, 
this opportunity to make any kind of comparison is 'tot only unusual but 
highly desirable. We were fortunate not only because we were able to make 
such comparisons but also because our pre-post sample turned out to be repre- 
sentative, in almost all respects, of the much larger group of teacher- 
participants who attended all our sample institutes. The results of this 
analysis will be discussed under The Samples in the section, WHAT WAS FOUND. 



Pretesting 



Pretesting was a two-stage transaction. The first stage was the dis- 
tribution of the Teacher's Mailbox, and the second, the attitude question- 
naires. Mailing was spaced so that the teacher-participant would have com- 
pleted or nearly completed the Teacher's Mailbox before he received the 
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questionnaires. This was planned primarily because the Teacher's Mailbox 
was a simulated test situation stressing a role-playing atmosphere, while 
the attitude inventories required honest, personal, se 1 f-appral sal s . 

Table 12 shows the pretest returns In terms of predictions and actual 
results. The table consists of three parts: results by subject matter area 
for the Teacher's Mailbox, results by subject area for the attitude scales, 
and results for the Teacher's Mailbox and attitude scales combined. This 
information is based on the number of tests initially sent to teacher- 
participants who agreed to participate in Project 2 and attended institutes 
in our sample. This number is 1,362 for all disciplines and becomes the 
total group (100 percent) on which we base our pretest predictions. 



Table 12 

Project 2 Pretest Returns 



Tests by 
Subject Matter 


Tests 

n 


Sent Out 

% 


Pred i cted 
n 


Returns 

% 


Actual 

n 


Returns 

% 


(1) Teacher's Mailbox 


1362 


100 


681 


50 


625 


ke 


Educational Media 


37^ 


100 


187 


50 


223 


60 


History 


355 


100 


177 


50 


127 


36 


Political Science 


258 


100 


129 


50 


m 


kk 


Teachers of Dis- 
advantaged Youth 


37^ 


100 


187 


50 


161 


k3 


(2) Att i tude Scales 


1362 


100 


681 


50 


7*t0 


5k 


Educat ional Media 


37^ 


100 


187 


50 


230 


61 


Hi story 


355 


100 


177 


50 


156 


kk 


Political Science 


259 


100 


129 


50 


139 


5k 


Teachers of Dis- 
advantaged Youth 


37^ 


100 


187 


50 


215 


57 


(l) and (2) Combined 


272k 


100 


1362 


50 


1365 


50 


Educat ional Med ia 


7^8 


100 


37k 


50 


k53 


61 


Hi story 


710 


100 


355 


50 


283 


ko 


Political Science 


518 


100 


259 


50 


253 


kS 


Teachers of Dis- 
advantaged Youth 


7k8 


100 


37 k 


50 


376 


50 
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Accordingly, we predicted that 50 percent or 681 teachers would return 
the Teacher's Mailbox materials. There were 46 percent or 625 teachers who 
returned Mailboxes, with history teachers returning the fewest proportion- 
ately (36 percent), and educational media participants providing the highest 
returns (60 percent). The average loss (decrement from expected 50 percent) 
of obtained pretest Mailboxes, approximately 2 percent, was due, we feel 
certain, to the many pressing obligations and commitments demanding a 
teacher's attention during the final weeks of the school year. The Teacher's 
Mailbox with its time-consuming requirements would, under those circumstances, 
hold little allure. These results meant that compa rlson of the pre-post 
sample with the total group of participants became even more imperative than 
we had contemplated earlier. 

On the other hand, the obtained attitude scale returns were, with one 
exception, all higher than our predictions. That is, 740 teachers In all 
four subject matter areas, or 54 percent, returned the attitude scales, 
whereas we predicted 681 teachers (50 percent) would so cooperate. Only 
history teachers (44 percent) did not surpass our 50 percent prediction. 

In all disciplines, however, more teachers were willing to complete the 
attitude inventories than the Teacher's Mailbox materials. This is, perhaps, 
to be expected, since we all find it easier and quicker to fill in blanks 
with check marks than discuss, in writing, items concerned with concepts 
and/or principles of a discipline. 

When the returns of both the Mailbox and attitude scales were combined, 
we found that our Actual Returns (50 percent) were the same as the Predicted 
Returns for all four areas (50 percent). Again, teachers at history insti- 
tutes gave the poorest returns of PROJECT IMPACT predictor instruments (40 
percent), while teachers who attended educational media institutes made the 
best showing (61 percent). 



General Reaction to Project 2 



Disappointed by the Mailbox returns, we were rewarded by the number and 
kinds of comments we received about the Mailbox itself. Generally, the 
response to Project 2 was one of overwhelming interest in the study and 
anticipation of a valid report for future use by teachers interested in 
improving their school communities. Listed below are just a few verbatim 
replies complimenting the project itself and those who designed it. The 
typical reaction to the Teacher's Mailbox by those who agreed to participate 
was, "I am most proud to be asked to participate." There was obviously 
initial interest shown, although many were forced to decline any reply to 
the Mailbox contents because of lack of time at the end of the academic year. 
Those who did follow through included many who thoroughly enjoyed the work, 
while others apparently needed more time for completing their answers in 
detai 1 . 

There were only about ten people who wrote to us expressing a desire 
not be be bothered at all with the materials because of various reasons. 

These included people who felt that the questions were ambiguous or not at 
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all connected with their professions. None of them, however, completely 
eliminated the idea of having a project such as this, but instead they gave 
their own suggestions as to how it should be run. Time was the major factor 
which i:^pt many from participating and most of them would have done so if 
the contents had been mailed earlier during the year. 



Some Verbatim Replies from Pretest Participants 

"Answering this mail has been a real learning experience. I am looking 
forward to the summer at . If PROJECT IMPACT is any indi- 

cation of things to come, this should prove to be a challenging summer. In 
closing, I congratulate you on the efficient manner in which the project was 
organ i zed ." 

"I think your letters were very challenging and well thought out. If 
I again may be of service to you, feel free to call on me." 

"Your approach is refreshing and your materials are excellent." 

"Even though I did not fill out the replies--the ideas were thought- 
provoking and have had an evaluative effect on me." 

"The work proved interesting and I feel better informed about educa- 
tional media because of all the investigating I had to do." 

"This experience has strengthened my desires to participate in the 

NDEA Institute at this summer, and given me the needed 

stimulus to champion the cause of disadvantaged youth. I'm exhausted, but 
it was worth it." 

"I really loved doing it and found it very thought-provoking — and that 
in itself is very worthwhile." 

"I was very enthusiastic about PROJECT IMPACT, and, in fact, I still 
am. I love the pretense. I did look at the Mailbox and felt that it would 
be a cha 1 lenge. " 

"I am grateful for having had the opportunity to 'air 1 some of my views. 
I believe it has helped me to solidify my own thinking on many issues. The 
school system within which I work does not usually provide such opportuni- 
ties." 



"I completely enjoyed participating in PROJECT IMPACT. I feel sure 
that much good will come from your work." 

"God bless your work." 

"This has been most enjoyabie, and I feel practical. I am sure we 
need more honest evaluation in education." 

"I congratulate you on the intensity of your study, and I thank you 
for endeavoring to aid in the most important work of teacher training." 
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Post testing 



Posttesting operations followed the identical design adopted for pre- 
testing. The Teacher's Mailboxes and the attitude questionnaires were again 
kept as separate and distinct transactions. The posttesting took place ap- 
proximately three months after the institute, in November and December. This 
schedule was selected to avoid, if possible, the pre-Christmas activities, 
but to allow sufficient opportunity for a teacher to transfer and translate 
into his daily teaching some of what was learned at his summer institute. 
Whether teachers can reasonably be expected to exhibit significant behavioral 
changes directly attributable to the institute experience, after only three 
months, is questionable. We were aware of the dangers of trying to measure 
impact so soon after the end of the institute program, but practical con- 
siderations forced us to adopt this schedule. 

Posttesting operations began with a letter informing teacher-participants 
that the posttests were to follow shortly. This letter was sent to all 
teacher-participants who had initially agreed to participate in Project 2. 

With more flexible deadlines for posttest returns, we tried to obtain as many 
returns as possible by sending follow-up reminders. Our first such communi- 
cation was a letter further explaining the need for full cooperation from all 
participants, if, indeed, Project 2 was to be a valid analysis. Our second 
notice was a postcard to remind participants of the approaching extension 
deadline, and the final reminder was a letter from the U. S. Office of Edu- 
cation. 

Table 13 shows the posttest returns in terms of our predictions and 
obtained results. Similar to the pretest table, Table 13 consists of three 
parts: results by subject matter area for: (1) the Teacher's Mailbox, (2) 

the attitude scales, and (3) both the Mailbox and attitude scales combined. 
This information is based on the number of pretests we obtained in the spring. 
This number is 625 for all disciplines, and becomes the total group (100 
percent) on which we based our posttest predictions. 

Originally, we predicted that **0 percent or 250 teachers would return 
the Teacher's Mailbox materials. We found 53 percent or 329 participants 
returned Mailboxes, with teachers of the disadvantaged returning the fewest 
(45 percent) and, surprisingly (in view of their ear 1 ier per formance) , 
history teachers providing the highest returns (58 percent). 

Similarly, the obtained altitude scale returns are al 1 higher than our 
predictions; that is, 342 teachers in all four subject matters, or 55 percent, 
returned the attitude scales, as against our prediction of only 250 teachers 
(40 percent). Again, we find history teachers providing the highest returns 
(58 percent), and those teaching the disadvantaged returning the fewest (46 
percent). This same picture holds when both the Mailbox and the attitude 
scale returns are combined. 

These data, we believe, show that our sampling scheme was an efficient 
one. The results also appear to conform to our hypothesis that lowered 
pretest returns resulted from the spring mailing schedule which took place 
during the feverish last weeks of the semester. 
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Table 13 



Project 2 Posttest Returns 



Tests by 
Subject Matter 


Tests 

n 


Sent Out 

% 


Predicted 

n 


Returns 

% 


Obtained Returns 
n % 


(1) Teacher's Mai Ibox 


625 


100 


250 


1*0 


329 


53 


Educational Media 


223 


100 


89 


1*0 


127 


57 


Hi story 


127 


100 


51 


1*0 


73 


58 


Political Science 


1 1** 


100 


1*6 


1*0 


57 


50 


Teachers of Dis- 














advantaged Youth 


1 61 


100 


61* 


1*0 


a 


1*5 


(2) Attitude Scales 


625 


100 


250 


1*0 


31*2 


55 


Educational Media 


223 


100 


89 


1*0 


128 


57 


Hi story 


127 


100 


51 


1*0 


71 * 


58 


Pol itical Science 


111* 


100 


1*6 


1*0 


66 


1*8 


Teachers of Dis- 














advantaged Youth 


161 


100 


61* 


1*0 


71* 


1*6 


(l) and (2) Combined 


1250 


100 


500 


1*0 


671 


51* 


Educational Media 


1*1*6 


100 


178 


1*0 


255 


57 


H i story 


251 * 


100 


102 


1*0 


li*7 


58 


Political Science 


228 


100 


92 


1*0 


123 


51* 


Teachers of Dis- 














advantaged Youth 


322 


100 


128 


' 1*0 


li*6 


1*5 



Thanking the Participants 



The PROJECT IMPACT staff felt a deep sense of gratitude to every teacher 
who worked on Project 2 instruments. We were always cognizant of the amount 
of effort required of participants, even though we were unable to reduce the 
work load without jeopardizing validity and reliability in evaluating insti- 
tute effectiveness. To express our appreciation, if only in a very small 
way, we included a note of gratitude in the final packet of attitude materials 
we forwarded. We wish we could have done more, but hopefully the intrinsic 
and altruistic reasons cited by the teachers themselves, for their cooperation 
and participation in Project 2, were the real rewards. 
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WHAT WAS FOUND 



The Samples 



General Comments 



Who attended NDEA summer institutes in a particular subject matter area? 
How best may we characterize the participants we worked with? How similar 
were they to other institute participants? How representative of teachers as 
a whole? Our concern with the nature of our sample was twofold. The first 
purpose sought to answer queries much like these, hoping to indicate the 
representativeness of our sample to the much larger group of teachers who 
attended NDEA summer institutes in 1 967 • The second purpose centered on 
obtaining appropriate and detailed information to describe our specific pre- 
post sample. 

Since populations can rarely be studied exhaustively, almost all research 
must depend on samples as a basis for describing population characteristics. 

We particularly needed sample-population comparisons because we used a mail 
approach where the proportion of respondents who return materials can be a 
crucial indication of the representativeness of the sample. 

A common, although not always available, technique for determining the 
representativeness of a sample is to compare one or more combinations of the 
respondents' qualities with those in the group who do not respond. More 
often than not, however, such evaluations are not possible since character- 
istics other than those being investigated are not measured. Project 2 was 
especially fortunate in being able to obtain this kind of information from 
the Application Record Cards. We selected 14 items for comparison, eliminat- 
ing those which were inapplicable or non-quant i f i able . These characteristics 
were grouped in two overall categories: (l) characteristics describing the 

institute teachers, and (2) those depicting the participants 1 home schools. 



Political Science 



0 
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The sample of 28* participants attending our 1967 political science 
summer institutes was very similar to the much larger group of 426 partici- 
pants. On those variables describing the personal characteristics of parti- 
cipants, we found that our sample did not differ significantly from the 
total group of participants on any variable. 



Our sample of political science participants did differ significantly, 
however, on one variable describing a respondent's home school; that is, the 
type of school. Three quarters of our sample were teachers from public 
schools, whereas over 90 percent of the institute participants taught at this 
type of school. The remaining 25 percent of our sample were from private 



*The number of respondents in each area does not always correspond with the 
obtained number in our samples because the Data Bank tape did not include 
information for all 1967 summer institute participants. 
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schools, especially church-related. We should point out that a second 
variable approached statistical significance. More teachers in the pre-post 
sample taught in schools with enrollments between 500 and 1,000 pupils, in 
contrast to the larger group, most of whom taught in schools with enrollments 
of over 1,000 students. 

Characteristics of the Sample . From additional demographic data, we 
get the following picture of political science participants. Men outnumbered 
women participants by more than four to one, no doubt a higher proportion of 
men than is usually found among teachers. The typical institute participant 
had taught an average of ten years with almost 40 percent reporting more than 
ten years. These experienced participants were also well educated, 50 percent 
having obtained master's degrees. Surprisingly, however, most of the master's, 
as well as the bachelor degrees, were not in political science, the subject 
field of the institute. The sample participants appeared similar to other 
groups in the attendance at institutes, about 70 percent never having attended 
an NDEA summer institute before. Finally, the vast majority of teachers in 
the sample (96 percent) taught at a single school, with the remainder having 
some form of split appointment. 

Characteristics of Participants' Home Schools. Participants appear to 
have come from most sections of the country with the midwestern and western 
states tending to be over-represented, as almost 60 percent of the group 
came from these two areas. In contrast, the southern states tend to be 
under-represented, fewer than 20 percent coming from this geographical region. 
Participants' school locations were distributed more or less according to 
those in the general population, with the largest group of participants com- 
ing from cities or towns of 2,500 to 50,000 population and the smallest number 
being from cities or towns of less than 2,500 or from rural areas. 

Over 50 percent of the participants were employed by schools with enroll- 
ments between 501 and 1,000 students. A sizable group (30 percent) were 
employed by large schools (enrollments greater than 1,000), with the smallest 
group of teachers (19 percent) teaching at small schools. Regardless of size 
or location of the schools, but in keeping with the level of the subject 
matter, over half of the institute participants taught at junior-senior high 
schools rather than elementary schools. 



History 

We were pleased to find that the personal characteristics of our pre- 
post sample of history participants closely represented the total group of 
teachers who attended U. S. history institutes during the summer of 1967- 
Only the variable of sex showed a statistically significant difference. Our 
pre-post sample contained a higher proportion of women than did the full 
sample of history institute participants. 

On characteristics describing the participants' home schools, we found 
additional differences between our pre-post sample and all other history 
participants. Proportionately, our sample contained almost twice as many 
teachers from private, church-related schools (elementary and secondary) than 
was descriptive of the total group. 



erIc 



39 



4S 



Characteristics of the Sample . In attempting to describe the history 
teachers in our sample, we found that they were youngish, wit!) 71 percent 
being under AO years of age. Related to this characteristic was the fact 
that approximately two-thirds of these teachers had taught for 10 years or 
less. We also found our pre-post sample to be well educated, with almost 
half having earned their M.A. degrees. While a majority of this group (58 
percent) had been history undergraduate majors, a very small minority re- 
ported previous institute attendance (11 percent). 

Character i st i cs of the Participants 1 Home Schools . The great majority 
of history teachers (79 percent) were f ronithe public school system. Over 
60 percent of the participants taught at the smaller schools, enrollments 
of 1,000 or fewer, and almost three-quarters were employed by junior and/or 
senior high schools. The midwestern and western geographical regions domi- 
nated, accounting for 62 percent of the sample, while 15 percent were from 
the northeastern portion of the country. The south appeared adequately 
represented, accounting for one-quarter of the sample. The school locations 
of our pre-post sample participants tended to be evenly distributed (between 
18 and 29 percent) among the four population categories ranging in size from 
large cities ov r 250,000 to centers of fewer than 2,500 people. 



Educational Media 



The pre-post sample of 115 media specialists attending NDEA institutes 
during the summer of 1967 appeared much like the larger media group of 972 
in attendance. Of the, eight variables describing participants 1 personal 
characteristics, our sample differed from the larger sample in only one 
respect--f iel d of graduate major. A somewhat larger percentage of the pre- 
post sample (13 vs. 8 percent) had graduate degrees in educational media. 

This percentage difference, however, is probably of little practical impor- 
tance since 9 A percent of each sample reported no undergraduate degrees in 
this area. 

On variables describing the participants 1 home schools, there was only 
one characteristic on which the samples differed; that is, the percent of 
black students enrolled in the participants' home schools. Both samples had 
a large majority of participants (about 90 percent) who were from schools 
which were predominantly white (75 " 100 percent). 

Characteristics of the Sample . The average age for the 115 media 
specialists in our sample was approximately A0, with 5 A percent of the group 
younger. As in the other three subject matter areas of our study, men out- 
numbered women, constituting almost two-thirds of our media pre-post sample. 
This group contained fairly experienced teachers, 56 percent reporting that 
they had taught for 11 years or more. A majority of the sample (53 percent) 
had earned their M.A. degrees prior to attending the 1967 media institutes, 

but, at most, 13 percent of these were in the field of educational media. 

Our sample is most likely composed of teachers who have transferred tcTthis 
area from other subject matter fields. 
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Characteristics of Participants 1 Home Schools . The large majority (75 
percent) of our p re-post sample was employed by schools in the midwestern 
or western region of the country, with the smallest percentage (5 percent) 
listing themselves as easterners. The northeast and southern sections were 
equally represented, each constituting about 10 percent of the sample. Most 
of the pre-post sample participants taught in the so-called smaller public 
schools of fewer than 1,000 students. And finally, our sample appeared 
equally divided between elementary and secondary positions, with each account- 
ing for b'i percent of the pre-post group. ( The remaining ]^% were included in 
categories other than the two mentioned*) 

Disadvantaged Youth 

A review of the biographical data for our pre-post sample of 72 partici- 
pants who attended institutes for teachers of the disadvantaged showed this 
group to be similar to the total sample of 1 ,068 institute participants. 

There were no statistically significant differences between the two samples 
for any of these personal characteristics. Similarly, on those variables 
describing the participants* home schools, we found only one statistically 
significant difference between the two samples, i.e., type of school where 
employed. Our pre-post sample was composed of a smaller percentage of 
teachers from public schools than was the sample of participants in general, 
thus containing a large proportion of teachers from private, church-related 
schools . 

Characteristics of the Sample . Unlike teachers in the other areas 
studied, this group of teachers was composed primarily of females, with over 
60 percent of the pre-post sample in this category. Their average age was 
about 39 years, with 59 percent being under ^0 years. The number of years 
spent teaching tended to be evenly distributed among the four categories 
ranging from over 15 years to under 5 years. A sizable percentage (38 
percent) had earned the M.A. degree, and some 10 percent indicated that their 
degrees were in the subject matter of the institute. Similar to the other 
subject matter areas, this pre-post sample also included a minority (11 
percent) who had previously attended a summer institute. Finally, almost all 
of our teachers (9^4 percent) taught at a single school, the remaining 6 
percent being employed either at the system level or at multiple schools. 

Characteristics of the Participants* Home Schools . The southern, mid- 
western, and western regions of the country dominated the geographical dis- 
tribution of schools as each of these accounted for approximately 30 percent 
of our sample. Only 8 percent of our teachers were from schools in the 
northeast, and only 3 percent taught in the eastern region. Over 70 percent 
of the participants taught at the smaller schools, enrollments of 1,000 or 
fewer. Also of interest is that almost two-thirds of this pre-post sample 
were teaching in elementary schools. Not surprisingly, 50 percent of the 
participants were employed by schools having black student enrollments of 
over one-quarter, while 36 percent were in schools where more than 75 percent 
of the students were black. Most of our sample also came from schools in the 
fairly populous cities or suburbs of over 50,000 population, and only 13 
percent were from towns of fewer than 2,500 or rural areas. 
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The Subject Matter Areas 



A repeated measures analysis of variance design was used and interpre- 
tations of the results made by both the PROJECT IMPACT research staff and 
the subject matter consultants. Although there were no significant changes 
in Mailbox performance resulting from attendance at summer institutes, these 
results do not necessarily indicate that the participants did not gain new 
insights about subject matter content and teaching strategies, but rather 
illustrate the difficulties so frequently encountered in seeking to show 
pre-post differences. The present approach contributed, however, a new 
dimension by permitting clinical assessments of the institute program for 
future restructur i ng. 

The Teacher's Mailbox allowed us to consider just such comparisons. 
Using: (l) the results of our analyses, (2) the written responses of the 

participants, and (3) the insights of discipline consultants, our subject 
matter specialists evaluated the summer institute program in terms of its 
effects on each teacher-participant, and the overall impact of individual 
institutes for improving the quality of teaching. Subject matter specialists 
were able to assess the styles and information used by the teachers in their 
simulated teaching procedures and judgments made for correcting deficiencies. 
These reports for the four subject matter areas follows. 



Political Science* 

As we pointed out previously, 1967 NDEA summer institutes in political 
science were held at 13 universities scattered across the United States. 

Each institute concentrated upon one or more Problems of American Democracy 
within the broad scope of political science, and each institute listed its 
own general as well as specific objectives. The objectives often reflected 
excessive optimism about what results could be obtained in a formal, limited 
institute setting. That is, they were, perhaps, often oriented toward an 
unrealistic expectation of what could be accomplished in the short span of 
a summer. 

Teacher' s Mai 1 box 

Table 14 presents the Teacher's Mailbox returns by letter. Average 
Quantitative and Qualitative Scores are given for our pre-post sample. Tests 
of significance indicate that there are no statistically significant differ- 
ences at the 5 percent level of significance between participants' achievement 
before and after attending the summer institutes. That is, our group of 
teachers did not improve their performance relative to the political science 
concepts and principles sampled by the Teacher's Mailbox and presumably dis- 
cussed at the 1967 summer institutes. 



*This section was prepared by James Barth and William McClure. 
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Average Quantitative and Qualitative Scores - 
Political Science Mailbox 



Letter No. 


Quantitative Scores 
Pretest Posttest 


Qu a 1 1 1 a 1 1 
Pretest 


Ive Scores 
Post test 


i* 


7.7 


7.2 


3.7 


3.3 


2 


8.1 


6.9 


3.3 


2.9 


3 


33.1 


27.1 


3.1 


2.9 


5 


23.7 


19.0 


3.3 


3.2 


6 


10.6 


10.A 


3.1 


2.9 


7 


11.1 


8.6 , 

/ 

y 


3.1 


2.8 


9 


17.6 


18.2 


3. A 


3.5 



* An average across topics 

We were not expecting dramatic results since many teachers had been 
Inadequately prepared to begin with In regard to contemporary approaches 
to the subject matter of political science. This Is not to say, however, 
that the Institutes have not had other beneficial results. The almost 
universal enthusiasm shown for the Institutes In the responses to Letter 9 
surely Indicates significant merit, particularly In terms of Improved 
perception of self and role. 

We shall consider the Mailbox Items In two groups, those emphas Izi ng 
content ( Letters 1, 2, 3 # 5 t and 6), and those emphasizing professionalism 
( Letters 4 °, 7, 8 0o , and 9 ). Nevertheless, each of the nine letters was 
very much a separate and distinct entity. Each letter had Its own objectives, 
and each required specific scoring, analysis, and evaluation. 

The Letters of Content 

Letter I. The participants were asked to discuss for the audience of 
a student newspaper two topics from a list of six. All topics were on 
selected problems of democracy, and Included the following: (I) politicians 

havte been known to buy votes; (2) presidents do not tell the public all 
the facts about wars; (3) Instances of police brutality are not unusual; 

(4) free speech Is sometimes limited by threats of reprisals; (5) the 
threat of force Is sometimes used to keep blacks from voting; and, (6) some 
public figures are far from perfect In their private lives. 

° LetteV requesting lists of equipment and. suppl les , was eliminated very 

early in our analyses because It was not amenable to what we believed was 
an appropriate evaluation. This letter did not structure the school situation 
sufficiently well to allow comparisons of equipment requests. 

00 Letter 8, requiring participants to fill out a Professional Activities 
Questionnaire, called for descriptive Information and did not lend Itself 
to objective scoring; hence It Is not included In the Quantitative and 
Qualitative Scores of the Teacher 1 s Mai 1 box. 



The intent of this letter was to see if the participants demonstrated 
three characteristics: ability to sense problems, ability to analyze 

process, and possession of an action-orientation toward finding solutions. 
Most respondents did not sense the existence of problems in the topics 
selected, or, if they did, they closed them off immediately through some 
standard answers or cliches. While a number of responses tended to be 
answer-oriented showing a reluctance to seek evidence, they were also typi- 
fied by a certain moral passivity. 

The most popular topic on the pre- and posttests was ‘’Presidents do not 
tell the public all the facts about wars. 11 This seemed to us a strange 
choice for teachers who are concerned with relevance, observable behavior, 
and inquiry. This topic deals with the one subject of the six most 
distant from the daily experiences of the respondents and the students whom 
they taught. The fact is that respondents rarely referred to personal knowl- 
edge or specific instances. Answers tended to deal with hypothetical cases, 
abstract concepts, and theoretical constructs, and to treat problems in 
terms of ideals and cliches. Perhaps they remembered that the local, the 
immediate, the real were likely to be more relevant. 

Letter 2. This letter asked respondents to criticize a civics quiz 
containing 25 items which students were to mark as "Acceptable" or "Insuf- 
ficient" statements about politics. Most of the 25 items included in the 
quiz were, in fact, cliches. The qualitative evaluation centered upon: 

(1) whether the respondent raised questions about proper use of the quiz; 

(2) whether he commented upon the quality of the quiz; and (3) whether he 
attempted to edit, change, or eliminate statements as a means of minimizing 
cl i ches. 

Both pre- and posttest performance were, on the whole, of disappointing 
quality. This was somewhat surprising in light of the fact that teachers 
are constantly engaged in testing of some sort. Perhaps the difficulty can 
be traced to the crude quality of the instrument. Many respondents simply 
rejected the quiz as an evaluative device, implying by their lack of critical 
work that the quiz was hopeless. Some suggested that the quiz might be used 
as a means of stimulating classroom discussion. This kind of response indi- 
cated that a respondent recognized a bad test when he saw one. It did not, 
however, illustrate his ability to construct tests. Neither did it indicate 
his grasp of the subject matter. 

Letter 3* This letter centered on survey design and content of the law 
pertaining to civil rights issues. Participants were instructed to cover 
three topics as part of a report on civil rights: (1) how to obtainable 
most accurate information about the civil rights law; (2) how to determine 
how concerned the community really is; and (3) how to find which local 
people are involved in making decisions. The responses were evaluated in 
terms of the list of sources of the law, the survey design, and the effort 
to identify local decision-makers. 

We noted three common approaches in answering this letter. First, most 
participants cited various information sources, balancing these sources to. 
avoid providing too "liberal 11 or too "conservative" a view on the civ^ 
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rights issue. Second, they demonstrated little familiarity with survey 
techniques. Finally, these respondents showed little acquaintance with the 
content and documentary sources of the civil rights law. We conclude that 
our respondents were probably not aware of the variety of approaches avail- 
able for carrying out rudimentary community studies most commonly associated 
with a study of political behavior. 



Letter 5* Letter 5 asked the participants to assist in rewriting a 
textbook to prepare teachers in the social science area. Unfortunately, 
these instructions were not clear in specifying whether the textbook under 
revision was to stress methods, whether it was to be a text on civics for 
teachers preparing to teach social science in secondary schools, or whether 
it was a textbook for use in the high school course itself. 

The objectives of Letter 5 centered on: 

(l) Educational Phi losophy . Did the respondents show their under- 
standing of the purposes of teaching civics? 



(2) Method . Did the participants discuss methods and techniques 
for teaching civics courses? 



(3) Content . Did the respondents comment on the content to be 
covered? 



A number of respondents did not recognize the appropriateness of a discussion 
of philosophy at this point. In fact, it seemed to us that some respondents 
preferred to steer away from controversy, although not necessarily from 
"cont rovers i al 11 issues. As for the "ideal 11 civics course, many respondents 
unfortunately failed to specify the means by which to achieve this goal. 

They simply ignored the problem, and those who addressed themselves to it 
were perfectly content to conceive their role solely in terms of maintaining 
current course structure. 



As to methods, respondents showed signs of awareness of the existence 
of a variety of teaching techniques which they identified as methods. 

The approaches they discussed were often not the most current ones. 

Although innovations were proposed, the emphasis still included the student 
as a passive learner. The child was to listen, he was to read, he was to 
memorize. 

Letter 6. In this letter, participants were expected to distinguish 
between the behavioral and institutional approaches for studying political 
phenomena. Adhering to our simulation format, this letter required the 
respondent to outline a telephone call to answer a mother's inquiries about 
her son's poor work in the behavioral ly-oriented civics course taught by 
our teacher, Richard Williams. 

Some respondents admitted quite frankly that they were unable to deal 
with the problem. Not all responses were evasive, but a number were con- 
cerned pr imar i 1 y wi th the counseling function. It seemed to us from reading 
Letter C that our respondents tended to see themselves as counselors rather 
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than subject and methods experts. Similarly, If our profession Is concerned 
with political behavior rather than structure, then we believe that the 
behavioral revolution" has not, as such, reached our respondents' class- 
rooms, either through undergraduate training or, as yet, through the sunnier 
Inst I tutes. 

The letters on professionalism 

ft seconc * P ar * Teacher's Mailbox covers an evaluation of Letters 

7, o, and 9. These three letters were concerned primarily with attitudes 
values, and professional teaching activities. The emphasis here was not only 
upon methods and techniques, but aimed at teachers spending time reflecting 
upon their professional experiences. 

Letter* 7* This item takes the form of a letter to a student who has 
just completed practice teaching and appears discouraged by the experience. 
Respondents were to answer several questions raised in this letter: 

(1) Don't the kids get on your nerves at times? 

(2) is it really worth the effort to try to tell them how exciting 
and important It is to know about our democratic process? 

(3) How do you feel about your students now that you have been 
teaching for almost three years? 

(4) Would you select the same career if you had your college work 
to do over? 

(5) Would you still prefer teaching at the high school level rather 
than at a college or university? 

Respondents to Letter 7 appeared to agree that the teaching profession 
has the distinctive and generally rewarding task in American society of 
"molding minds." In contrast, statements indicated little interest in 
encouraging the student to develop his own abilities and qualities. The 
problem was generally not seen in terms of what was good for the student, 
what was consistent with his wishes or psychic needs, or whether he needed 
to identify and solve problems. The acceptable criterion adopted by many 
participants was M what is good for society." 

Nevertheless, the evidence based on our sample of participants does 
question just how much knowledge the respondents had about teaching citizen- 
ship. Our analysis also pointed out that some respondents showed little 
change relative either to their teaching practices or In their verbal command 
of new concepts. We can only remind ourselves that assimilation of ideas, 
even under the best conditions, Is a slow process. During the regular school 
year, most teachers are allowed little time for reading, for thought, for 
intellectual growth and development. Generally, they do not read the profes- 
sional journals. To a great extent, therefore, they are dependent upon the 
popular press for their ideas, thus their tendency to emphasize the topical, 
the current, and often, the superficial. A five-to-six hour teaching schedule 




46 




leaves little time for experimentation. Architectural and political limita- 
tions must surely discourage it even further. In this light, the summer 
institutes take on a new importance, not because they lead to fundamental 
redesign of teaching, but because of their effect upon morale and percep- 
tion. The institutes show that at last someone is concerned about teaching. 

As the teachers themselves, almost without exception, point out, the insti- 
tutes permit teachers to meet and talk with other teachers. The opportunity 
to get away from the teacher's role with all its urgencies, pressures to 
conform, and intellectual and personal frustrations, and to retreat for a 
time in the company of kindred souls may yet prove to be the necessary foun- 
dation for educational reform. The brief institute programs may have pro- 
vided this personally beneficial respite. 

Practice teaching, the teacher's supervised apprenticeship in the pro- 
fession, had been, obviously, a traumatic experience for most of our respon- 
dents. It came as no surprise to our 5^ teachers that their make-believe 
friend, Carol, was depressed after her six weeks of practice teaching and 
was thinking of leaving teaching forever. One called it a case of "beginning 
teachers* jitters." Others described it in terms of morale and tended to see 
it as a continuing problem, extending even beyond the practice teaching 
period. 

Letter 8. At this point, the Teacher's Mailbox offered a "Professional 
Activities Questionnaire" covering the three areas of professional journals 
and professional activities. Respondents were asked to check appropriate 
items, with the scoring system for Letter 8 consisting of frequency counts 
by item and category. 

It was obvious from early returns that our teacher-participants were 
not joiners of professional associations. About one-third of the respondents 
belonged to the local, state, and national chapters of the Council for the 
Social Studies. They belonged to few, if any, history, political science, 
sociology, or other social science associations. 

For the most part, our respondents did not read the professional 
journals. The one-third who received the National Council of Studies 
Journal stated they read it, but almost none of the others said they did. 
Furthermore, our results indicated that participants rarely subscribed to 
other scholarly journals. 

The major professional activities engaged in by our sample of teachers 
included attendance at conferences, workshops, and lectures. Four semi" 
professional activities most frequently engaged in by our participants, 
activities not a part of their regular teaching loads, are in decreasing 
frequency order: (1) set up trip to county or city government offices; 

(2) chaperoned Saturday trip to county or city government; (3) supplied list 
of civics books to order for local public library; and (4) spoke with stu- 
dents about civics teaching as a career. 

Letter 9. This letter asked participants to comment about their expec- 
tations of attending summer institutes (pretest) and their reactions to the 
institutes they had attended (posttest). Evaluation was directed to three 
points: (1) a discussion of how the institute would broaden the participant's 
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experience; (2) statements on the importance of the institute as a means of 
working out the participant's subject matter concerns and problems; and 
(3) information for updating and stimulating wider understandings in political 
science content. 

The reasons most commonly cited in the pretest by the participants for 
attending institutes were that institutes: (1) permitted teachers to meet 

and talk with other teachers; (2) provided factual information; (3) presented 
new approaches to teaching; and (4) provided for professional growth. Post- 
test responses were generally the same, with the following variations. 
Participants: (1) now praised the institutes because they made it possible 

to work with experts; (2) commented favorably about the methods of presenting 
subject matter topics; and (3) were less concerned with graduate credit than 
they had been in the pretest. Examination of the responses to Letter 9 
clearly indicated that institutes were well received by the respondents. 

There were few negative comments on institutes as such or on any aspect of a 
specific institute and its operation. This result reaffirmed the findings 
of many other studies on the widespread, enthusiastic acceptance or summer 
institutes. 

Attitude Assessment 

The pretest and posttest results for the attitude questionnaire are 
presented In Table 15* 



Table 15 

PROJECT IMPACT Attitude and Opinion Survey Results - 
Political Science Participants 



Sub test 


Pretest 


Posttest 


Ch i 1 d-centerf i Approach 


35.0 


36.0 


Teaching-centered Approach 


17.4 


16.4 


Profess ional i sm 


20.3 


20.3 


Effectiveness as a Teacher 


12. 1 


12.0 


Trad i t ional- Emergent Approach 


80.3 


84.7 



The higher scale score for "emergent" value structures classified 
teacher-participants as likely to emphasize sociability, a relativistic moral 
attitude, group conformity, and enjoyment of the present.. That is, the 
participants appeared to shift, at least in direction, from a "traditional" 
value structure characterized by respectability, thrift, self-denial, indi- 
vidualism, and orientation toward the future. The concern clearly reflected 
a child-centered approach. If this emphasis follows through in teachers' 
classroom behavior, then the institute process has, indeed, had an affective 
impact. 
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The operation of Institutes, with opportunities to observe effective 
teaching in action, listen to those with educational expertise, and exchange 
ideas with colleagues, may, in fact, be key factors encouraging even greater 
pupil concern by attending participants. What we are saying is that the keen 
enthusiasm so often expressed by teacher-participants for NDEA summer insti- 
tutes may, in reality, be a reflection of the opportunities to engage in 
just such pursuits. 

The results of the other instrument for assessing teacher personality, 
The A1 Iport-Vernon-Lindzey Study of Values are shown in Table 16. It is 
important to remember, before interpreting AVL score differences, that this 
instrument indicates only the relative importance of values for a given 
individual, rather than their absolute value or magnitude. Inspection of 
Table 16 shows no change in the pre-post scale scores for our sample of 
respondents. That is, none of the scale differences is statistically sig- 
n i f i cant. 



Table 16 

A1 1 port-Vernon-Lindzey Study of Values - 
Pol i t ical Science 



Scale 


Pretesr 


Posttest 


Theoretical (Concern for Truth) 


40. 9 


40.0 


Economic (Utilitarian Inclination) 


36.6 


37.2 


Aesthetic (Appreciation of Form and Harmony) 


- 36.1 


37.1 


Social (Feeling for Others) 


42. i 


42.6 


Political (Position on the Use of Power) 


44.0 


43.9 


Religious (Cosmic Orientation) 


40. 1 


39.1 



History* 

Accepting as its basic premise that the primary purpose of the 1967 NDEA 
summer institute program was to improve the knowledge, attitudes, and teaching 
skills of the teacher-participants, Project 2 sought to assess the impact of 
the institutes in U. S. history on those who attended them. As will be seen 
from the following discussion of the results, the data produced by these 
devices attested to certain strengths and weaknesses in the history insti- 
tutes, and also generated a number of suggestions for the planning of future 
American history summer institutes. 



*This section was prepared by Henry G. Waltmain, Donald L. Parman, and Floyd J. 
Fithian. 




The Teacher's Mailbox 



Although they overlapped at several pdints , the nine Mailbox letters 
fell Into three general categories: Letters 3, 5, and 6 focused on subject 

matter and historical concepts; Letters 1 , 2, and A emphasized i nst ructional 
strategies; and Letters 7, 8, and 9 stressed professionalism. Letter 4, which 
asked the participants to list the types of teaching aids and equipment they 
believed necessary for efficient Instruction, was discarded and is not covered 
in this analysis because it was widely misinterpreted by the respondents. 

Average scores by letter are presented In Table 17.* Analysis of these 
data indicated that there were no statistically significant differences 
between pre- and pos t- I nst I tute performance. In view of the differences 
among the Teacher's Mailbox items, each letter will be considered indepen- 
dent ly. 



Table 17 

Average Quantitative and Qualitative Scores - 
HI story Mai 1 box 



Letter No. 


Quanti tat ive 
Pretest 


Scores 
Post test 


Qual i tat ive 
Pretest 


Scores 

Posttest 


i 


36.0 


32 . l 


3.5 


2.9 


2 


30.3 


28. 4 


3.5 


3.3 


3 


26.2 


26.0 


3.3 


3.2 


5 


20.9 


21.7 


3.4 


3.5 


6 


19-7 


18. 4 


3.4 


2.9 


7 


19-2 


16.5 


3.3 


3.0 


9 


22.6 


24.3 


3.4 


3.3 



Subject matter and historical concepts 

Letter 1. The first item in this series, devised as a memorandum from 
Mr. Williams, the school principal, asked the respondents to describe what 
they regarded as an ideal course in American history. They were urged to do 
so on the assumption that they had access to unlimited funds and had full 
control over personnel, curriculum, and materials. Essentially, this letter 
attempted to determine whether the history Institutes had given the partici- 
pants a fuller appreciation of what was needed to develop an effective course 
In their field. The participants' comments revealed, for example, that they 
were generally reticent in discussing the particulars of staffing, scheduling, 



*Scores for Letter 8 have been omitted since a different system was followed 
to score this item. 



and administering a special course, both in the spring and in the fall. 
Conversely, whether anticipating or reflecting upon the NDEA sessions, they 
readily expressed their views on the relative merits of topical, chronologi- 
cal, or problem-solving courses, on the necessity of close contact between 
teachers and students, and on the utility of certain classroom materials and 
teaching aids. 

Letter 2. The second item in the Mailbox series was a letter from the 
addressee's superintendent requesting information on the development of a 
U, S. history course for the benefit of a visiting Canadian educator who was 
planning to introduce such a course in his country's schools. Basically, it 
asked Mr. Williams what he would present as a "typical" unit in an eleventh- 
grade American history course. In responding to this inquiry, the teacher 
was asked to comment on reading, reporting and writing assignments, class- 
room techniques, the use of special equipment, and student evaluation. 

The results here seemed to suggest that some aspects of the teachers' 
unit planning were affected by the institutes. A comparison of their spring 
and fall comments implies that there may have been some shift in emphasis 
which did not show up on the overall statistics. Before attending the NDEA 
programs, the respondents commonly mentioned the need for group activities 
by their students, but were rather indefinite about how this need was to be 
fulfilled. In the fall, they were more exacting in their discussion of this 
matter. 

Letter 3. The next item was a memorandum testing the recipient's knowl- 
edge of historical method. The school principal, Mr. Grant, asked for a 
brief outline of "the process by which historians work," stating that he 
needed this information for a debate with a school board member who was enthu 
siastic about the educational value of the "hard" sciences, but skeptical of 
the utility of the social sciences, [n answering, the teacher was urged to 
expand upon the argument that history had a unique, generally accepted method 
"based on certain rules of evidence and logical assumptions." 

In general, the answers to this item point to these conclusions: 

(1) the teachers who attended the institutes were generally con- 
vinced of the importance of what they were teaching; 

(2) many of them found it hard to articulate the rules of evidence 
historians used, to compare these procedures with those utilized 
in science, or to explicate the relationship between teaching 
and research; and 

(3) most of the respondents did not change or clarify their views 
on these matters after taking part in the institutes. 

Letter 5* Written as a memorandum from the school principal, this item 
was a request for what amounted to a critical book review of one of the 
classics in historical interpretation* Having recently illustrated his 
acumen in this sort of exercise in an after-dinner speech, Mr, Williams was 
asked to spend twenty minutes analyzing one of the great works of Frederick 
Jackson Turner, Vernon Parr ington, Charles Beard, Walter Webb, or any other 



work of his own choosing. The major purpose of this communication was to 
determine whether the institutes had Increased the participants' understand- 
ing and appreciation of interpretive history. If so, this could be regarded 
as an important educational contribution by the NDEA programs. 

An examination of the respondents' remarks suggests that at least three 
factors may have accounted for the resulting posttest scores. First, for 
academic and personal reasons, the participants and their institute mentors 
shared a special interest In the relationship between contemporary and tradi- 
tional scholarship in their field. To some extent , perhaps, they mirrored 
the growing inquisitiveness and sophistication of the younger generation, 
not content with knowing the who's, what's, when's, and how's of American 
development, but anxious to know why things happened and why they were and 
are important. Second, the study of basic historical interpretations was, 
perhaps, not only one of the more compelling aspects of the teachers' work 
at the institutes; it was also more concrete and manageable than some other 
assignments. The kind of information involved was conducive to short-term 
"gap-filling" and was easily transferrable. And third, in comparison with 
some of the more general questions posed in other letters, the summarization 
and evaluation of a single great interpretive work was an exercise which 
could be handled well in a brief reply. 

Letter 6. The general subject of this letter was the meaning or philos- 
ophy of history. Attached to the letter were eight definitions of the pur- 
pose and value of history, quoted from notable historians with various 
backgrounds and viewpoints. The participants were asked to do the following: 
(1) select the definition which they preferred; (2) justify their selection; 
and (3) verify the authorship of each definition. The main purpose of the 
letter was to determine what influence the history institutes had on the 
participants' conception of history. In this regard, a teacher's ability to 
defend a particular definition of history was considered more important than 
the definition itself. 

From the definitions they chose and the justifications they gave, it 
seemed to us that few of the respondents had a sophisticated, carefully 
reasoned, current philosophy of history. Instead, they had a rather vague 
and unsystematic set of ideas and emotions with respect to their discipline. 

Professional i sm 



Letter 7* This letter covered many of the routine, but serious per- 
plexities of teaching history in the public schools. The writer of the 
letter had just completed student teaching, had encountered several problems, 
and was asking an experienced teacher how he met these difficulties. Specif- 
ically, the participants were quizzed in regard to the following: (l) getting 

along with students, (2) overcoming student apathy in required courses like 
history, (3) making history exciting, and (4) remaining in public school 
teaching rather than moving to college teaching. 

The message most clearly conveyed by this inquiry was that the history 
teachers were generally satisfied with their present positions and were not 
actively considering other jobs, particularly at the college level. In ex- 
plaining their contentment, they frequently cited the rewards of working 
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with young people, including the pleasures of getting to know them personally 
and watching them mature. Several respondents stated that they were not 
attracted to college positions because of the "publish or perish" pressure 
that existed there. Even those who expressed an interest in college teaching 
were not highly critical of their current assignments. 

Letter 8. Letter 8 in the Teacher's Mailbox asked the participants to 
fill out a Professional Activities Questionnaire before and after attending 
the institutes. The questionnaire called for information on their member- 
ship in national, state, and local organizations, on their subscription to 
and use of professional journals, and on their involvement in activities 
related to their teaching assignments. 

In general, the statistical findings of this inquiry paint a disappoint- 
ing picture of the institutes 1 influence on the history teachers 1 professional 
interests. The total figures on the number of professional affiliations, 
current and anticipated, on the number of journals subscribed to, to be 
ordered, and read, and on the number of activities attended, engaged in t or 
under consideration showed no posttest increases. Clearly, history teachers 
were neither joiners of professional societies nor readers of scholarly pub- 
lications in their field. 

Finally, a tabulation of the participants' replies with respect to their 
academi c endeavors outside the classroom produced the most involvement in 
three activities: (1) library or book display work, (2) field trips, and 

(3) career planning. These results held for both pre- and posttesting. 

Letter 9- In the pretest version of this letter, the writer, who indi- 
cated that he was undecided about taking a summer job, going to summer 
school, or applying for an NDEA institute, asked the would-be participants 
for advice on what to do. Thus, in effect, the respondents were asked what 
they expected to gain from their institutes. In the reworded posttest com- 
munication, the same writer attempted to find out whether or not the NDEA 
summer institute programs had lived up to the teachers' expectations. Hence, 
Letter 9 solicited some basic reactions on the value, as well as the strengths 
and weaknesses, of the history summer sessions. 

The reasons the teachers most frequently gave for wanting to attend the 
institutes were practical in nature. By and large, they hoped to learn more 
about their field and exchange ideas with other teachers. In addition, many 
of them wished to improve their teaching methods and were anxious to know 
more about techniques involving student performance. Some were also motivated 
by the prospect of graduate credit, financial rewards, and academic stimulation. 

After attending the institutes, the participants generally indicated the 
programs had fulfilled their expectations, in fact, their posttest responses 
cited some benefits which had not been anticipated. Once more, increased 
factual knowledge, contacts with other teachers, and new teaching methods re- 
ceived primary attention. Many respondents also stressed their appreciation 
for the academic stimulation and opportunity to read and study. On the whole, 
moreover, much of the pretest emphasis upon nonacademic matters (stipend, 
travel, graduate credit) gave way to an increased recognition of the value of 
professional training, exposure to specialists, and other academic activities. 
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A 1 1 i tudes and Value s 



The results of the A1 1 port-Vernon-Lindzey Scale of Values and the PROJECT 
IMPACT Teacher Attitude and Opinion Survey Indicated that the history programs 
did not appear to modify the teacher-participants 1 values or attitudes. This 
was to be expected In view of the shortness of the summer sessions and the 
relative Inflexibility of these personal factors* 

The findings, shown in Table 18, invite at least three observations 
about the institutes and those who attended them. First, all the pre- and 
posttest scores fell within the normal upper and lower limits for this kind 
of analysis, although the political and economic figures were on the high 
and low side respectively. Second, the teachers showed no pre- and post- 
test scale score differences. Third, the order of importance which the 
participants assigned to these six values was not changed by the institutes. 
Ranging from greatest to least emphasis, that order remained: political, 

religious, social, theoret ical aesthetic, and economic. 



Table 18 

A1 Iport-Vernon-Llndzey Study of Values - History 



Scale 


Pretest 


Posttest 


Theoretical (Concern for Truth) 


39.5 


39.3 


Economic (Utilitarian Inclination) 


36.5 


37.5 


Aesthetic (Appreciation of Form and Harmony) 


38.3 


38.5 


Social (Feeling for Others) 


39.9 


40.0 


Political (Position on the Use of Power) 


kk. 3 


43.9 


Religious (Cosmic Orientation) 


b).b 


40.8 



Meanwhile, the returns on the PROJECT IMPACT Teacher Attitude and 
Opinion Survey also followed a predictable course for all five scales. 

These results, presented in Table 19, show that the fall scores on the child- 
centered, professional, effective teacher, and traditional-emergent items 
which treated these matters were no different from the pre- i nst i tute scores. 
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Table 19 



PROJECT IMPACT Teacher Attitude and Opinion Survey - 

HI story 



Scale 


Pretest 


Posttest 


Ch I 1 d-Centered 


33.3 


32.9 


Teach i ng- Centered 


19.7 


18.3 


Professional! srri 


20.7 


20.5 


Effective Teacher 


12. 1 


12.2 


Traditional-Emergent Value 


82.6 


82.5 



Educational Media* 

Updating the teachers' knowledge of educational technology requires a 
systematic approach to determining instructional problems and effecting 
feasible solutions to them. For this purpose, a new breed of professionals 
called media specialists has emerged since World War II to assist teachers 
in instructional improvement. Such specialists are relatively f ew in number . 
and their levels of competence have typically been inadequate. The NDEA 
media institutes were thus designed to increase both the number and compe- 
tence of media personnel in the elementary and secondary schools of the 
nation. 

A large majority of the media institutes worked with begi nners--teachers 
with some media responsibilities in individual school buildings or teachers 
desiring to assume such responsibilities. Librarians also took part in these 
institutes to help prepare themselves to assume some media responsibilities 
in the school library. There were, in addition to beginner institutes, 
numerous advanced institutes for the more experienced media specialists and 
a few institutes for teacher trainers. Thus, there was a wide range of 
experience and competence represented among the participants attending media 
inst i tutes. 

Quite clearly, a single institute, however effective, could handle only 
a few of the competencies required by a we 1 1 -qua 1 i f ied media specialist. 

The evaluation effort described in this report, however, faced dealing with a 
heterogeneous sample population including both beginners and experienced 
media personnel as well as numerous librarians with limited backgrounds in 
teaching, and some classroom teachers, with a single test instrument. 

The Teacher's Mailbox 

In qeneral, the differences between pre- and posttest Quantitative 
and Qualitative Scores on the nine Teacher ‘s Mailbox 



*This section was prepared by Robert Kline. 
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Letters were consistently small and not statistically significant. These 
results are presented in Table 20. We also observed that on the pretests 
a number of respondents frankly acknowledged, at the start, their inability 
to answer the questions asked. 



Table 20 

Average Quantitative and Qualitative Scores 
Educational Media Mailbox 



Letter No. 


Quant i tat ive 
Pretest 


Scores 

Posttest 


Qual i tat ive 
Pretes t 


Scores 
Post test 


i 


17.2 


18.2 


2.3 


2.5 


2 


10.2 


10.3 


2.8 


2.7 


3 


11.2 


12.1 


2.4 


2.6 


4 


12.9 


13.2 


3.1 


3.1 


5 


14.1 


14.1 


3.1 


3-1 


6 


69.9 


75.5 


3-7 


4.0 


7 


13.8 


13.9 


2.8 


2.8 


9 


10.3 


lt.l 


3-3 


3.5 



The letters 

Letter 1. The first item in the Teacher's Mailbox was a memorandum 
addressed to "All Media Coordinators" informing them that the Superintendent 
of Schools wished recommendations from each school on equipment they wished 
to have for copying purposes and for making transparencies. 

An examination of the mean scores for Letter 1 indicated that our 
institute participants did not appear to be well informed about either 
duplicating machines or their uses in schools before they attended summer 
institutes. Our post- i nst i tute results, however, did not provide a more 
optimistic picture. These results suggest that a number of the institute 
programs probably did not include copying machines as part of their programs, 
or that they dealt with this equipment on a very limited basis. Supporting 
this observation is the fact that almost all of the respondents referred 
primarily to the machine with which they were familiar in their own particular 
school. 

Letter 2. Letter 2 in the Teacher's Mailbox was a memorandum from the 
principal to his media coordinator. The principal indicated that he was to 
address the local Rotary Club and that he had decided to focus his remarks 
on the school's audiovisual program, primarily because questions concerning 
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expenditures for non-book instructional materials had appeared in the press. 
The purpose of this Mailbox Letter was to get the respondents to: (1) dis- 

cuss the rationale of their commitments to a sound audiovisual program; 

(2) list the major points that form the foundations for their viewpoints; 
and (3) give specific psychological and theoretical bases of learning which 
apply to their points of view. 

The responses offered to questions in this letter seemed to imply that 
more attention to the basic foundations underlying audiovisual programs in 
schools was called for in many institutes. The phrase "psychological and 
theoretical bases of learning" seemed to cause the respondents some diffi- 
culty. There was strong indication in the written replies that they were 
not familiar with these bases of learning. A superficial reason frequently 
cited for using media, for example, was "a picture is worth a thousand 
words" or "ten thousand words." 

Letter 3* Mailbox Letter 3 was a memorandum from the principal to the 
audiovisual director. In this memorandum, attention was focused on instruc- 
tional procedures, and, specifically, on how one analyzes content, including 
the identification of learning tasks and learning problems. The letter also 
referred to a student guide for previewing the films which the director had 
indicated he would like to prepare. The purpose of this memorandum, as an 
item in the Teacher's Mailbox, was to discover what the participants knew 
and understood about instructional development procedures. 

Since the respondents were also asked to describe a student guide for 
previewing films and to explain how decisions regarding its development were 
determined, it appeared probable that attention was, in some cases, diverted 
from procedures for analyzing content and identifying learning tasks. It 
seemed to us that the responses given to this letter ,ind icated that, whatever 
had been the expectations of the Mailbox development, little was done with 
either instructional development (a sophisticated process about which rela- 
tively little learning can be expected--particularly by beginners in a short 
summer institute) or with student film guides. If this conclusion does 
follow, then Letter 3 may not, in all fairness, have been a "good" item. 

Letter A. Letter 4 in the Teacher's Mailbox was a letter from a media 
coordinator in a particular school addressed to the media director of the 
school system. This letter revealed that the director had presented a science 
lesson to the district teachers which the writer had found stimulating and 
helpful, and which made him realize that he could and should give the teachers 
in his school more leadership. He then asked the director if he could give 
him a step-by-step description of how he works with a teacher on the develop- 
ment of a specific lesson. 

The intent of this Mailbox item was to give the respondents an oppor- 
tunity to: (l) set forth their criteria and procedures for the evaluation 

of media, and (2) indicate their approaches and techniques for assisting 
tochers in making decisions rege : ng the selection and use of materials in 
specific teaching/learning situ; 
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Participant responses indicated some exposure in the media institutes 
to the instructional development procedure. 1 Since the diffusion and dis- 
semination of instructional devel opment i n format i on was a primary obligation 
of the NDEA Institutes, this question was an appropriate and important one. 

One reason indicated by the scorers for the relatively low posttest scores 
was that many respondents dealt with methods for distributing equipment 
rather than with the selection and utilization of media for improving the 
quality of instruction. 

Letter* 5. The topic considered in Letter 5 of the Teacher's Mailbox 
required respondents to describe: (l) how a media coordinator helps teachers 

locate instructional materials in the school, the system center, and other 
media centers, and (2) how he helps them locate commerical materials and 
equipment not available in any of these centers. 

Respondents, in answering this letter, typically referred to their own 
school situations rather than to adequate or appropriate situations such as 
those that were most likely presented in the institutes. Some respondents 
also indicated strong biases and narrow views of alternatives in this area 
which were apparently not altered by institute experiences. 

Letter* 6. A memorandum from a state media director to all media co- 
ordinators plus a "Media Characteristics Matrix" constituted the sixth item 
of the Teacher's Mailbox. The intention of Letter 6 was to elicit informa- 
tion concerning characteristics unique to various types of meoia to determine 
how well informed the participants were with respect to essential knowledge 
about the "tools" of the media field. 

In responding to the checklist, the institute participants tended to 
localize their answers. In part, this may have been due to the fact that 
numerous items on the checklist called for opinions. For example, one item 
was "textbooks are easy to preview." Obviously, if textbooks are readily 
available in a given situation, it is easy to examine them, but if textbooks 
are not readily available, then the situation is just the opposite. Similar- 
ly, to say that previewing films is difficult, or to say that it is easy, is 
entirely situational, depending on the availability of a film and/or pro- 
jector. 

Letter* 7. Mailbox Letter 7 was a rather long letter from a teacher who 
had just spent four weeks working with teachers on the development of media 
for some South American units. The teacher was downhearted and disillusioned 
about her efforts to improve instruction through the use of media and frankly 
admitted her doubts about the ef f i c iency of the new instructional technologies. 
The purpose of Letter 7 was patently to determine the rationale institute 
participants would present in the face of a direct criticism of media, and 
their use in teaching and learning. 

We anticipated that the responses to this letter would show that parti- 
cipants, through their institute experiences, had developed a rationale and 
a methodology for working with teachers who have reservations about the use 
of media in instruction. 
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Why this did not happen is not evident* According to scorers, mipst 
participants wrote prol i f I cal ly , but did not thoroughly develop their* 
reasons for implementing media in the learning process. In partial explana- 
tion of this, the scorers surmised that the superfluous information included 
in Letter 7 may have caused some respondents to take a tangential course in 
formulating their replies. Generally, t fie responses fell in two categories: 
(1) answers such as, "we'll discuss this later over a cup of coffee, "ior 
"things will get better if you stick with it," and (2) generalities sjuch as 
"sensory experiences are more lasting than verbal exper iences, 11 or othjer 
statements equating educational media with visual material. ; 

Letter 8. Letter 8 in the Teacher's Mailbox was a questionnaire con- 
ducted on the professional activities of all teachers in the county. ] 

The objective of this Mailbox item was to obtain a measure of the: 
respondents' degree of professional commitment. An examination of the: 
responses to this questionnaire raises some questions, however, about f:he i r 
validity. The inconsistencies between the pretest and posttest responses 
on actual and planned membership in national and/or state professional \ media 
associations seemed to suggest that the respondents read and respondedUo 
the questionnaire in what appears to be an almost whimsical fashion, fcn 
consequence, the data were questionable and it is difficult to draw general- 
izations from them. 

In short, the nature of professional development as a consequence ;of 
institute attendance was not readily apparent from responses to the question 
naire. It seems that participation in an institute was sufficient, at 
minimum, to foster an initial generation of increased professionalism. - It 
is probably also correct to assume that in-depth changes in this aspect of 
professional behavior require more time to develop than the interval between 
the institute and the posttest administration, and it is, therefore, not 
surprising that our instruments did not measure such a change. 

Letter 9. Two versions of a letter from a prospective institute parti- 
cipant to a committed participant comprise the final item in the Teacher's 
Mailbox, In the letter the participant was given before he attended an 
institute, he was asked to list the beneficial features or results he ex- 
pected to find at the institute. In the letter he received after attending 
an institute, he was asked to indicate if the institute had met his expecta- 
tions, and what its good and bad points were. 

The responses to this letter were much the same as responses to this 
question for other disciplines. That is, most of the participants were very 
favorable toward the summer programs both before and after attendance. 
Favorable comments centered around staff, other participants, media equip- 
ment, and access to relevant literature. The few criticisms made primarily 
concerned general organization of specific institutes and the poor rapport 
between staff and participants for specific programs. 

Attitude Assessment 



We had anticipated that the institutes might have affected the attitudes 
and values of the participants. The results of the PROJECT IMPACT Attitude 
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and Opinion Survey, as well as the Study of Values , reveal no pre-post dif- 
ferences on the attitude dimensions measured by these instruments. For com- 
parison purposes with the other subject matter areas, we have compiled these 
resul ts i n Table 21 . 



Table 21 

PROJECT IMPACT Attitude and Opinion Survey - 
Educational Media 



Subtest 


Pretest 


Posttest 


Ch i Id-Centered 


37.0 


37.3 


Professional ism 


20.1 


20.8 


T each i ng-Cen t e red 


17.9 


17.3 


Effective Teacher 


11.8 


11.9 


Traditional ly- Emergent 


84.7 


84.3 



Table 22 

Allport-Vernon-Li ndzey Study of Values 
Educational Media 



Scale 


Pretest 


Posttest 


Theoretical (Concern for Truth) 


39.9 


41.0 


Economic (Utilitarian Inclination) 


39.9 


60.6 


Aesthetic (Appreciation of Form and Harmony) 


. 38.6 


38.5 


Social (Feeling for Others) 


38.2 


37.8 


Political (Position on the Use of Power) 


39.7 


38.5 


Religious (Cosmic Orientation) 


44.2 


, 42.5 



Training the Teachers of Disadvantaged Youth* 

This section deals with those institutes which were held in the summer 
of 1967 for the teachers of disadvantaged youth, conducted at 27 universities 
and colleges. Although the legislation under which the institutes were 
funded, Title XI of the NDEA was categorical in nature and specified the 
particular subject area under which teachers were to be trained, the exten- 
sion of the legislation to include "disadvantaged youth" moved training into 



*This section was prepared by Daniel Bernd and Ray Johnston 
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an area that could hardly be said to represent given disciplines in the same 
way that history institutes, English institutes, or political science insti- 
tutes represented the disciplines specified by the law. There not being a 
discipline known as “disadvantaged youth," the legislative mandate to "in- 
crease the subject matter competency of teachers" could not be met by focus- 
ing upon the structures of a single subject, by dealing with defined problems 
within a particular discipline (such as Problems in American Democracy), or 
by increasing the identification with the professional group organized around 
a given discipline. Therefore, the question of target, or participant, was 
much more overriding in the institutes for disadvantaged youth than it was 
in the institutes which reflected particular subject areas or disciplines. 

The instructional programs of the DY institutes were wide-ranging, and dealt 
with such areas as the basic research and the basic theory from several 
disciplines on the effects of impoverishment, needed structural reforms in 
the schools, appropriate educational media and instructional aids, and ways 
of securing support for program development. Even the concept of "disad- 
vantaged youth 11 varied among the 27 institutes. The disadvantages were 
described variously as mental disturbances, economic deprivation, ethnic 
differences, racial bias, and the special problems of the "slow learner." 
Thus, the DY institutes were organized around the problems of the ultimate 
cl ientele— the elementary and secondary children who were, in one way or 
another, educationally deprived. 

It should be noted that, with few exceptions, the institutes were 
planned and developed by academics from Schools of Education and Social Work. 
The institutes themselves, except one, were housed in Colleges or Departments 
of Education. In 1967, very few institutes of higher education were conduct- 
ing interdisciplinary programs dealing with the problems of disadvantaged 
youth. The basic theories, the conceptual frameworks, came from such disci- 
plines as sociology, anthropology, political science, and psychology, but at 
the onset of the federally supported programs, it must be remembered that 
their staffs were drawn mostly from those interested in applying theory 
rather than developing it, and that very few training programs for such 
teachers existed. The institutes, then, were concerned not so much with 
retraining or upgrading a particular defined class of teachers, but in 
creating a new class. 

Some of the complexities arising out of these facts were reflected in 
the proposals. The institute participants varied widely in background, 
interests, and occupational role. Concerned with a ripple, or multiplier 
effect, some institutes attempted to recruit administrators and teacher- 
trainers as well as teachers. Most programs (22) selected either 30 or kO 
participants, four had 50 or 60, and one attempted to train 80. 

Even granting these variations, and granting the lack of a definable 
interdisciplinary subject matter, the DY institutes still had enough simi- 
larities to justify their being classified under the same rubric. All of 
the DY institutes considered themselves as agents of social reform, particu- 
larly by wishing to increase the number of professionals involved with DY. 
Most institutes relied on consultants from the relevant social sciences to 
present cross-disciplinary alternatives. Every institute made an effort to 
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select its participants on the basis of background, experience, and motivation. 
Every Institute was required to bridge the gap between theory and practice by 
giving participants field experience in, as the Office of Education Guidelines 
stated, those areas where live the “environmentally disadvantaged." 

Of the preceding, there is little doubt that the sense of mission, of 
the teacher and teacher-trainer as change-agent dealing with a particularly 
defined group of students, is the constant in the variable DY institutes. 

The U. S. Office of Education Guidelines assumed that certain psychological 
and social characteristics most often associated with poverty could be changed 
through knowledge and through attitudinal training, and these assumptions were 
reflected in the individual institute programs. 

The evaluative question is, did the DY programs, in fact, train their 
participants to better deal with the problems of teaching disadvantaged 
children? Did the participants become more "professional" after attending 
a DY institute? Can they exhibit the skills and transmit the knowledge the 
institutes were designed to give them? PROJECT IMPACT, of course, is the 
study designed to help answer those questions, and to help pinpoint the diffi- 
culties involved In asking them. 

The results shown by the instruments used by the IMPACT staff (the 
Teacher's Mailbox, the Attitude and Opinion Survey, and the Study of Values), 
when applied to DY institutes are much the same as the results shown by these 
instruments when applied to institutes in history, political science, and 
educational media. That is, the participants consider the institutes to be 
worthwhile and positive experiences, but we cannot show that their post- 
institute level of competencies tests any higher than their pre- inst i tute 
level. The teachers do not show any significant change in their values as a 
result of attending a summer institute, nor do their tested attitudes show 
much difference (Attitude and Opinion Survey). Given simulated tasks, they 
did not score significantly different after the institutes than before. 

But to say that the test scores do not show significant differences is 
not to say that the test instruments were not informative. Particularly in 
the area of disadvantaged youth does the process of constructing the test 
clarify the structures of the subject, those concepts and skills which ar^ 
considered to be relevant in the teaching of these teachers. And the test 
results, because of the questions asked, reveal the difficulties involved in 
achieving the overall goal of the institutes — social change. An analysis of 
the Teacher's Mailbox as used with participants in DY institutes is especially 
useful in what It suggests about the gap between the objectives of training 
programs centered in higher education and resultant teacher behavior in the 
schools. 

The Teacher's Mailbox 

Whatever the results of the test, and whatever the efficiencies of simu- 
lation devices in yielding pre- and posttest results in short-term training 
programs, the Mailbox does reveal what the field Itself considers to be im- 
portant. The failures of the teachers to demonstrate that they learned what 
the experts in the field thought they should know gives some rather clear hints 
of how training programs should be changed in order to face the realities of 
the school situations. The following analysis of the nine letters to which 
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the DY participants were asked to respond indicates what some of those reali- 
ties may be. But anilysis of the Teacher’s Mailbox results does raise a 
wealth. of questions for people in the field itself and suggests possible di- 
rections for further investigations. The Mailbox results are presented in 
Table 22. 



Table 22 

Quantitative and Qualitative Mailbox Scores - 
Teachers of Disadvantaged Youth 



Letter No. 


Quantitative 

Pretest 


Scores 
Pos ttest 


Qual i tat i ve 
Pretest 


Scores 

Posttest 


i 


22 


20 


3.2 


3.5 


2 


18 


18 


2.7 


3.1 


3 


20 


20 


3.1 


3.3 


4 


24 


23 


3-3 


3.1 


5 


11 


10 


3.1 


2.8 


6 


40 


37 


3.2 


3.4 


7 


12 


11 


3.6 


3-4 


9 


20 


24 


3.1 


3-5 



The letters 

Letter h The first letter to Richard Williams of Lindenwood School 
in Linden City is from another teacher in another disadvantaged school and 
asks for advice in dealing with "restlessness and constant interruption by 
the kids." The respondent is asked also to indicate what psychological 
theories are relevant to the problem. 

The assumption behind the question here is that competent teachers in 
school situations of the kind described will move toward a more open and 
less punitive attitude, will not stereotype the behavior, and will be able 
to consider the behavior in the light of some stated psychological theory. 

Letter 2. In the second letter, Richard Williams is asked by the 
chairman of an "Ad Hoc Committee for Violence Control" to consider whether 
a policeman should be placed in the school hallways in order to control 
rowdiness, uproar, and student violence against teachers. 

This letter was designed to elicit responses that indicated to what 
extent the institute participant was prepared in the techniques of conflict 
analysis and negotiation. The responses were scored 1 to 25 on a continuum. 
The acceptance of police control as correct and proper scored low, and sug- 
gestions for program analysis of root causes scored high. The assumption 
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was that such problems were better handled, by student and negotiation-oriented 
Individuals, and that control-oriented, non-negotiators, were not likely to 
handle the problem well. Most teachers, however structured, apparently main- 
tain nor -negotiable areas where control is deemed necessary. From the point 
of view of planning, educational systems should probably take cognizance of 
this desire for stability. 

Letter 3. This letter is concerned with means and methods. Williams 
is asked by a college professor preparing a methods text to describe the 
strategies and ph i losoph les behind an ideal course. The central problem is 
described as one of capturing the interest and imagination of the turned-off 
disadvantaged youth. 

The letter clearly calls for responses that go beyond standard approaches, 
and is intended to discover the participants* knowledge of the pedagogical 
techniques, educational media, and experimental methodology appropriate in 
disadvantaged schools. 

Letter 4. The fourth letter attempted to get at the question of profes- 
sional sophistication and awareness. Richard Williams is asked, in this 
letter, to supply information to a foreign educator on current programs for 
the disadvantaged and to comment on their objectives, substance, and effec- 
t iveness. 

The assumption behind the letter is that high scores are indications of 
professional commitment to the field and of the methodology and structures 
of the d i sc i pi ine. 

Both Letters 3 and 4 attempted to get at the questions of appropriate 
course content and of knowledge of the field. The scores suggest that the 
university faculty members* expectation of public school teachers are not 
met. An analysis of the results suggests that teachers remain structure 
and course content-oriented (in keeping with most of their educational ex- 
perience), and are reluctant to adopt new pedagogical methods and to innovate 
and experiment. Further, participants maintain a . provincial rather than a 
cosmopolitan attitude to available program solutions to the problems of dis- 
advantaged youth. These results suggest that it is difficult for a retrain- 
ing program to change the styles of teachers in the face of the immediate 
influences upon them. That Is, faced with an educational system that is 
structured toward course content, the teachers will continue to respond to 
that system, whatever they have learned about Its deficiencies. 

Letter 5. In this letter, the principal of JJndenwood School asks 
Richard Williams for specific knowledge of appropriate tests for disadvantaged 
children, and requests a “brief description of how we go about or should go 
-about evaluating the disadvantaged students in our school...* 1 ' 

The responses to this letter were scored to determine the participants': 
(1) knowledge of recent findings regarding reliability and validity questions 
on standardized tests, (2) knowledge of recent developments of non-verbal 
evaluative techniques, and 
testing and evaluation. 
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(3) knowledge of new trends and programs in 
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This letter attempted to measure the Impact of the institutes upon the 
participants' knowledge of: (l) the shortcomings and inequities of stan- 

dardized testing, and of (2) alternative means of student evaluation. The 
results suggest the power of testing as a status and control instrument. 

Given a very firm and clear belief on the part of the experts that 
present evaluation and test instruments are not only inadequate in dealing 
with disadvantaged children, but are positively destructive, and given a 
group of highly motivated participants, and given the fact there are new 
programs and new trends in testing and evaluation, why do the posttest re- 
sults fail to show that the teachers learned anything that they were willing 
to apply? 

The results from this letter are particularly interesting in what they 
suggest about the problems of educational reform. Given the task which 
requires them to indicate what they actually would do, in a real situation, 
the teachers revert to conservatism, to the old ways. Testing is a way of 
exerting power, of having influence in the school, and the teachers did not 
feel supported enough by their actual school situations to try to effect 
changes. 

Letter 6 . Here, Richard Williams is asked to assume the role of educa- 
tional planner and to design an ideal program, both for buildings and facili 
ties and for curriculum. The task set was to ignore the obstacles at 
Lindenwood School and to "imagine you are setting up ideal arrangements for 
handling disadvantaged youth at Erewhon School." 

Perfect participant response in the design of the new school would 
include the involvement of pupils and a broad community population in basic 
decision-making, would relate the plant facilities to the community, would 
specify the curriculum and the media, and would include an upgraded system 
which specifies an individually oriented teaching methodology across all 
courses and all fields. 

Letter 7 . In this letter, a prospective teacher asks Richard Williams 
how he really feels "about dedicating yourself to such a cause, now that 
you've been teaching them day after day for almost three years." The pros- 
pective teacher is discouraged at her own pract ice-teach i ng experience and 
couches her concern in emotionally loaded terms. 

Rather than scoring the participants for personal dedication alone, the 
responses were also judged in terms of how well they moved away from moral iz 
ing toward giving professional answers. This letter was designed less to 
measure "improvement" and more to discover how the teacher's conception of 
his role changed as a result of the institute. 

This letter presumed that attendance at an institute would affect the 
kind of recruiting advice one would give, and that answers would move away 
from moralizing to professionalism* The results suggest that the disad- 
vantaged youth movement is still in its ear 1 y organ i zat i ona I stages, and has 
not yet moved into the stage where it attracts individuals who view the work 
as a matter of professionalism, although it is tending, as a movement, in 
that direction. 
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The manner in which the participants performed the tasks of the Teacher's 
Mailbox rather strongly suggests that changes are very difficult to bring 
about in education If we Ignore the total system within which teachers work. 
The author of changes and Innovations often finds himself alone, ignored by 
his colleagues and resisted by his superiors. Any program for the training 
and retraining of teachers must take cognizance of the fact that the pres- 
sures against change often appear to the teacher as being very much greater 
than the pressures for change. It should not be surprising that the data 
from the Teacher's Mailbox revealed this conservatism in action. 

Letter 8. Letter 8 is designed to discover the participant's concep- 
tion of his role as a professional in affiliation with other professionals. 
Richard Williams Is asked what organizations he belongs to, what journals 
he subscribes to and reads, what workshops and meetings he has attended, and 
what his future plans are. He is also asked for "personal comments concern- 
ing professionalism of teachers. .. # M 

The task set assumes that a measure of a teacher's commitment and under- 
i standing of his field is in his professional activities. 

Letter 9* The crucial question asked in this letter is whether the 
participant would recommend applying to an institute to another teacher. 

| Richard Williams is asked to jot down a list of the "beneficial features or 

results" of the Institute. (The pretest letter varied slightly in form from 
| the posttest letter. The first asked for expectations and the other inquired 

Into resul ts. ) 

The scoring of this letter was designed to reveal not only the commit- 
ment and attitude before the Institute experience, but also how it changed, 
if it did. 

In summary, if any one conclusion may be fairly drawn from the Mailbox 
as applied to teachers of disadvantaged youth, it is that training and re- 
training programs for teachers alone are not a sufficient condition for 
educational reform and that school systems as systems must be considered in 
the design of such programs. 

Attitude Assessment 

Tables 2^ and 25 show the results of the attitude questions for teachers 
attending DY Institutes. These data, as in the previous subject matter areas, 
Indicate no pre-post differences. 
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Table 24 



Attitude and Opinion Survey - 
Teachers of Disadvantaged Youth 



Subtest 


Pretest 




Posttest 


Chi Id-Centered 


38 




39 


Teach i ng- Centered 


19 




16 


Profess ional ism 


20 




20 


Effective Teacher Rating 


12 




12 


Traditional-Emergent Values 


85 




86 



Table 25 






Study of Values - 






Teachers of Disadvantaged 


Youth 




Scale 


Pretest 


Posttest 


Theoretical (Concern for Truth) 


39.6 


38.8 


Economic (Utilitarian Inclination) 


39.3 


39.8 


Aesthetic (Appreciation of Form and Harmony) 


37.7 


37.7 


Social (Feeling for Others) 


41.1 


h \ .9 


Political (Position on the Use of Power) 


38.5 


39.1 


Religious (Cosmic Orientation) 


i*3.8 


h 2.7 



Some Generalizations 

The greatest success of the DY Institutes comes from the fact that by 
participating In the PROJECT IMPACT study, it appears that DY is a field of 
study, with its own conceptual frameworks and methodologies. The construc- 
tion of the items in the Teacher's Mailbox, and particularly the construction 
of the scoring keys, focused the attention of the professionals centered in 
various disciplines onto the problem of describing a coherent structure of 
concepts, theories, skills, methodologies, and evaluative instruments by which 
the particular programs could be measured and compared. 

This phenomenon suggests, perhaps, that more important than the ques- 
tion of what participants learned or did not learn is that there was indeed 
something to be learned, that disadvantaged youth is a describ a Me subject . 
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(There is little doubt, for example, that the necessity to describe its 
subject matter through constructing a device similar to the Teacher's Mailbox 
would be a useful and revealing exercise for the field of English.) 

Evaluation as an informative process, a process of clarification and 
suggestion, is much more important than evaluation as mere testing. Our real 
concern should not be to answer the simplistic question of whether the DY 
institutes were "good" or "effective," but rather the question of what we . 
learned from the evaluation instruments that indicate what changes should be 
made to improve the training of teachers. The IMPACT study is particularly 
rich in what it suggests about the education of denied children. It suggests 
that we know a good deal, and it suggests that teachers positively want to be 
professionals, and value their association with the professionals in higher 
education (Letter 9), but it also suggests that making the school an agent of 
social change is a very difficult and complex undertaking. If they have suc- 
ceeded in revealing the nature of the problem, then it can be said that the 
DY Institutes and the IMPACT study have done their jobs. 
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PROJECT 3* 



THE PURPOSE 



The major focus of Project 3 was to determine the impact of the NDEA 
summer institutes on the colleges and un i versi t ies wh i ch hosted them* The 
NDEA institute program encouraged university faculty members to work direct- 
ly with elementary and secondary school teachers. For many professors, this 
was a first experience. It was hoped that this interaction among subject 
matter academicians, department of education staff members, and teachers who 
attended summer institutes, would establish a new milieu dedicated to improv- 
ing present educational procedures. 

A third pilot study, Project 3 elected to sample a small number of 
institute directors and faculty members, seeking to learn of any implementa- 
tion of institute practices in their normal professional activities as a 
measure of Impact* The PROJECT IMPACT staff recognized from the start that 
such changes would probably be of an individual nature and, therefore, neither 
easily measured nor observed. We believed, therefore, that an in-depth per- 
sonal interview with a staff member offered the most efficient and effective 
method for gathering the appropriate information. This approach permitted 
an interviewer the necessary flexibility to identify basic issues for better 
understanding and interpretation of the meaning of impact. 



THE METHODOLOGY 



The Interview Schedule 



To insure that we obtained answers to important and relevant questions, 
we decided to use a structured interview schedule. This procedure of asking 
all participants the same questions maximized the number and comparability 
of responses while minimizing interviewer bias and intuitiveness. The 
PROJECT IMPACT staff, the interviewer, and our consultants then set about to 
develop the interview schedule based on the original guidelines established 
by NDEA Title XI for conducting summer institutes. The final items selected 
were based on the possible effects of the institute program on the subject 
matter specialists, the educators, and the participating teachers. 

Accepting the pilot status of Project 3» we continued to include only 
four subject matter areas for study. These were the same four disciplines 
selected in Project 2: educational media, history, political science, and 



*This section was prepared by John Curry. 
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teachers of disadvantaged youth. Two cons iderat i ons guided our selection 
of institutes: (1) university enrollment (we tried to choose three insti- 
tutes in each discipline, according to university size: small, medium, and 

large ); and (2) the number of years that these universities had hosted 
institutes, with two years required as a minimum. Our total sample, then, 
consisted of twelve institutes, three per subject area. 

All interviews were conducted individually, with the interviewer travel- 
ing to each of the twelve colleges or universities to talk with as many 
institute directors, institute staff members, and related faculty members as 
possible. The plan to speak with professors who had not participated in the 
NDEA summer institute was adopted to provide as complete an evaluation of 
institute effectiveness as possible. Difficulties of scheduling and meeting 
arrangements precluded obtaining the same number and types of interviews at 
all twelve host institutions. 

Interviews were tape recorded where possible, with approximately 90 
percent of the interviews so handled. All interviewing was done during the 
fall semester following the 1967 summer institutes. 



Scoring the Tapes 



Most of the taped Interviews were brief, averaging about thirty minutes. 
Much of the information served to describe the questions asked by our inter- 
viewer. The scoring procedure adopted was a content analysis. 

To perform the content analysis, each of two independent scorers lis- 
tened to each interview in its entirety, recording all questions and answers, 
but deleting responses which seemed unrelated to our meaning of impact in 
this study. A single scoring key was found satisfactory for all four disci- 
plines since interviewing variability was greater within a content area than 
among the subject matters. 

Scoring was a tabulation of the frequencies of responses to each ques- 
tion across people; no individual was given a score of any kind, nor was any 
attempt made to weight the responses according to any predetermined defini- 
tion of impact. Two raters independently scored each tape. 

Comparison of the frequency distribution obtained by the two scorers 
revealed low agreement on a few of the answers and only moderate agreement 
on many of the answers. It should be noted that inter-rater agreement was 
not obtained quantitatively, but was determined roughly by comparing the two 
sets of scoring sheets for each question. 

There are several explanations for the generally low correspondence 
between the two scorers: (1) phrasing of questions was not constant across 

interviews; (2) wordiness and irrelevant rambling obscured the answers to 
some questions; (3) some of those interviewed contradicted themselves; (A) 
not all answers and comments could be handled by the scoring instrument; 
and (5) some of the discrepancies were scoring artifacts which resulted from 
differences in the wording of answers. 



THE INTERVIEW RESULTS 



General Comments 



Several comments were made during the interviews which were specific 
to individual institutes and were not, therefore, included in the regular 
scoring. A brief discussion of these comments by subject matter aret 
follows. 



Educational Media 

Staff members from educational media institutes appeared favorable 
toward the institute experience, and they reported more overall change than 
did the respondents from any other discipline. One director stated that: 

The whole field of audiovisual professionalism has grown 
as a result of these institutes. While there have been 
quite a few big milestones in the development of media in 
education in the last 25 years, probably the Title XI pro- 
gram in media is the most significant milestone that I 
have seen. 

Other directors felt that one important way in which the institute programs 
helped was in the discovery of new talent, particularly in the public schools. 
Some of these institute participants have become media leaders within their 
own states, and some have become known nationally. Similarly, some respon- 
dents stated that universities were now beginning to emphasize their educa- 
tional media programs, and that this interest was probably due to the insti- 
tutes, because former participants were known to be pressuring universities 
to offer these programs or improve existing ones. 



History 

Although the positive reports from the history institute faculty members 
tended to be guarded, with some negative comments, the most common positive 
comment concerned the pleasant, rewarding nature of the institute for every 
member of the institute staff. Some stated that the institute had awakened 
the faculty to its responsibility in improving teacher education. 

One of the positive opinions voiced by a non-staff member of a history 
department was that the institute had helped the image of the department 
throughout the university. Similarly, another non-staff member from the 
education department stated that there had been great impact in terms of 
generating enthusiasm among members of the departments and that he, personally, 
had become interested in teaching in an institute. On the negative side, one 
director expressed the opinion that some institutes merely repeat the medi- 
ocrity of regular summer school programs, having little impact on their 
universities. 
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Pol i t ical Science 



Several political science staff members commented that the institute 
program had been a pleasant experience, especially in that it permitted 
contact with school teachers. In addition, it produced or increased the 
feeling of responsibility which their department now felt toward these 
teachers. The general environment of institutes was cited as a factor in 
encouraging the positive perception of participants toward different groups 
of people. 

Generally, this area appeared to show the least amount of change, 
although some departments did modify their regular academic programs in 
some ways. 



Teachers of Disadvantaged Youth 



In general, respondents from this area were favorable toward the insti- 
tute experience, some being highly complimentary and none voicing any strong 
complaints. An especially interesting report concerned the improvement in 
racial attitudes which was believed to be brought about by the integrated 
institute program. A staff member stated that improved attitudes were found 
in his own department, as evidenced by the hiring of more blacks and the 
admission of black students, and in the county school administration where, 
he said, the integration process progressed as much as ten years. Once the 
public school administrators saw how well integration worked in the institute 
program, they appeared more receptive to having blacks on the staff and in 
the student body. A member of another institute staff felt that his insti- 
tute had also been successful in changing the attitudes of the participants 
toward disadvantaged students, particularly, black students. 



Questions and Answers 



The major results of this project consist of a variety of answers to 
the questions presented to institute directors and staff members in our 
sample. The complete analysis of our findings is shown in Tables 26, 27, 
and 28. The tables list each question asked, with frequency returns for 
the topics used in our content analysis. These results do offer interesting 
indications of institute effectiveness. In view, however, of (l) the small 
number of answers to each question, and (2) the nature of the procedure, we 
caution against any broad generalizations of the present findings, based on 
twelve summer institutes, to all NDEA institutes. 

The following pages will present discussion of answers to each of the 
major questions asked in the interviews. These questions have been homogen- 
eously grouped according to content, and are followed by an appropriate 
table. 
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Academic Practices 



I. Have there been any changes in curriculum? One-fourth of the 
respondents indicated that there had been curriculum changes, and about half 
of those interviewed stated that such changes were planned or proposed, 
although not implemented. Major changes or proposed changes included areas 
offered for minor study and curricula for prospective teachers. Respondents 
from history and disadvantaged youth institutes accounted for most of the 
affirmations of actual curriculum changes. 

I I . Have there been any changes (increases or plans to change) in writ- 
ing or research? More than half of the respondents replied negatively to 
this question, indicating either that they had no knowledge of any new writ- 
ing or research or that the staff had already been doing much in this area 
before the institute. The remaining staff members indicated that new writing 
or research had been initiated, or at least that there existed an increased 
interest in research as a result of the institute. In some cases, new re- 
search or increased interest in research was derived directly from the insti- 
tute experience, e.g., follow-up studies of institute participants were 
conducted by some staff members or directors. In other cases, the institute 
had reaffirmed interests which had existed for a considerable period of time. 
Contrariwise, one history department member stated that his writing had been 
interrupted by the Institute. 

III. Are there any differences in your teaching strategies or those of 
your col leagues? About two-thirds of those interviewed stated that the 
institute had resulted in changes or planned changes in teaching methods, 
particularly in terms of course content, use of textbooks and audiovisuals, 
and student teaching. Some staff members indicated that they were using 
more paperbacks as supplementary reading or that they were insisting that 
students be a little more aware of available books in the field, while others 
spoke of less specific changes, such as being jarred out of routine and 
regimented course presentations. Several faculty members spoke specifically 
of changes in advanced courses as well as changes in basic courses. 

Certain changes in academic practices may have been easily identified 
by the faculty as having resulted directly from the institute experience, 
but other changes were probably difficult to relate to this summer program. 
The present data-gather i ng scheme is assumed to have elicited only informa- 
tion which concerned changes directly attributable to the summer institutes. 
There is the possibility, however, that the affirmative answers of some 
participants were based on conjecture about tangential relations of certain 
academic practices to the institute program. 



Effects on Members of the Departments 

I . What was the most significant gain or impact that you, personally , 
received from the institute? A majority of the institute faculty answered 
that the major personal benefit they derived from their institute involve- 
ment was either the enjoyable contact with the teachers who attended the 
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Frequency of Responses to Questions in the Interview Schedule 

Academic Practices 
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Total percentages may add up to 99% or 10l%. 



institutes or the increased awareness of teachers' problems and needs. The 
participants were described by one history staff member as being more inter- 
ested in asking why and learning to understand the material than in memoriz- 
ing facts in order to get grades. More than one-third of those interviewed 
felt that their major personal benefit had resulted from their different 
institute teaching experience, which some respondents described as a cause 
for their improved instructional methods. 

I I . Were there any changes in your attitudes or those of staff members? 
For example, were there any changes in attitudes toward public school 
teacher sT Almost 80 percent ofthe respondents stated thaT the i ns t i tute 
experience had brought about increased awareness of teachers' problems and 
greater interest in teacher education. Most of the remaining respondents 
indicated either that their attitudes had recently changed, but that they 
could not definitely attribute the change to the institute, or that they had 
already been concerned with teachers' problems before the institute. Several 
of those interviewed indicated that they had felt little, if any, concern 
for teachers prior to the institute, but that they now viewed the teachers 
as their professional colleagues, with tremendous problems. 

The question of attitude changes was probably the most difficult one 
for which to obtain reliable responses, because of the problems associated 
with determining attitude changes, and because of difficulty in attributing 
perceived changes to the institute program. However, in the case of faculty 
members who had worked neither with teachers nor with a department of educa- 
tion prior to the institute, an increase, at least in awareness of teachers' 
problems, may have been fairly easy to identify and assign to the institute 
experience. 

III. Have there been any differences in intra- and/or inter-departmental 
relat ionsT About AO percent of the replies affirmed that the institute had 
stimulated both closer inter-departmental and i nt r a-departmenta I relation- 
ships. Changes in personal relationships among departments were usually 
easy to attribute to the institute. Some individuals who met initially while 
planning or conducting the institute continued to work together during the 
ensuing academic year. Increased or improved i nt ra-departmenta I associations 
were probably somewhat difficult to attribute to the institute except in the 
case of marked changes in personal relationships, like the situation described 
by a history non-staff member who stated that his institute colleagues con- 
stantly discussed their experiences. 

I V . Are you or the staff working differently (or more concerned) with 
students, graduate or undergraduate, who are planning to become teachers? 
Almost half of the respondents indicated that they were using new content 
or teaching methods in working with prospective teachers, and about one-third 
said that they believed themselves to possess increased sensitivity to the 
needs of prospective teachers. They believed that the institutes had caused 
them to realize that these students have a use for information which was dif- 
ferent from theirs, that is, primarily for teaching rather than for research 
and writing. 
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V. Do you find yourself or other staff members working differently 
with teachers in the public schools? Fewer than 30 percent of the respon- 
dents stated that there had been an increase or change in the work which 
faculty members of their universities were doing in the public schools, 
although many of the remaining staff .-members said, that there had already 
been much of this kind of activity prior to the institute. There were sub- 
stantial differences among the four disci pi ines. The highest percentage of 
Indicated change was educational media (50 percent), whereas no one inter- 
viewed from the political science institutes had observed such activity. 

V I . Are you or are other staff members working differently with teacher , 
or other professional groups? Almost half of the respondents answered that 
they were working differently with teacher (or other) professional groups, 
and more than half of the remaining respondents stated that they had already 
had numerous associations with these groups before the institute. There 
were again marked differences among the disciplines. Sixty percent or more 
of those interviewed from history and educational media answered that they 
were working differently, or more, with teacher professional groups, whereas 
only one-third of the respondents from institutes for teachers of disadvan- 
taged youth so stated. Again, no one of the political science staffs responded 
pos i t I ve ly . 



Institute Participants 

I . Have any of the participants gone on, or back, to graduate school? 
The respondents reported here that they believed numerous participants had 
done post-institute graduate work or at least applied for such study. It 
should be remembered that the pos i t ive responses to this question did not 
necessarily mean that the participants were returning for graduate work at 
the un i vers i ty wh i ch hosted the institute, although this appeared to be the 
case in at least a few instances. We considered the responses to this ques' 
tion as more indicative of institute impact on the teacher-participants, 
and, therefore, on the teaching profession, than on the universities which 
conducted them. Further, it is difficult to evaluate the net effect of 
creating, with the participants, a greater interest in subject matter. 

I I . Is there any program of follow-up on the institute participants , 
or is there any feedback from the participants? Follow-up was usually in 
the form of newsletters, visits to schools, or informal meetings. Like the 
previous question, we felt that data on feedback and follow-up do not neces 
sarily provide specific information relating the effect of the institute on 
the host institution. We would assume, however, that a university depart- 
ment which is following up its institute with any aid or assistance to the 
participants is demonstrati ng that it definitely has an interest in working 
with school teachers. 



Criticisms and Suggestions 

The final question asked of those who were interviewed concerned comments 
or criticisms which they might have about USOE's handling of the institute 
program. The most frequently advanced suggestion by both directors and staff 
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members was that a plan for systematic follow-up of participants should be 
included in every proposal, or that there should be a division in the U. S. 
Office of Education which would visit school systems. These plans would 
encourage working relationships w,i th' pr i net pa 1 s , admi n i st rators , and teachers 
in order to implement the suggestions of those teachers who had attended 
institutes. Some of the respondents stated that PROJECT IMPACT was the type 
of follow-up which they felt should be conducted. 

Another frequent criticism was that there should be long-range funding 
of institutes in order to facilitate planning. One political science staff 
member stated: 

The time has come to get [the institute program] off the 
erratic, jerky, unplanned, unsystematic basis. ...and start 
thinking in terms of long-range proposals. Institutions 
can‘t plan...Jhere should be some goals; there should be 
some systematic planning. 

There was considerable comment about proposal writing and other nego- 
tiating which are necessary before universities can receive funding for 
conducting institutes. The lag time for approval of funding was cited as a 
problem in that it is difficult to get commitments from faculty and to struc- 
ture the program months ahead. A history staff member suggested that the 
U. S. Office of Education have a pre-packaged institute program so that 
faculty members would not have to spend their time writing a proposal for 
funding. This policy would, of course, place the burden of innovation on 
USOE rather than on the host institution. 

Another topic discussed concerned selection of participants. An inter- 
esting difference of opinion here, related to the ability levels of those 
who should attend summer institutes. Some faculty members believed that 
only "top echelon" should be considered, that is, those with advanced degrees, 
in addition to several years of teaching experience. Other respondents, how- 
‘ ever, preferred a diverse group of participants, differing widely in their 

ability and experiential levels; while a small number of those Interviewed 
urged selecting participants on the basis of real "need." 
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SOME CONCLUSIONS 



What Indeed have been the “lessons 11 of the NDEA summer insitutute? 

While findings of the three PROJECT IMPACT studies did not answer all the 
questions, we believe that various results should be of interest to those 
who (1) shape and administer USOE policies, (2) direct and train teachers, 

(3) apply for and attend teacher-training programs, and (4) are charged 
with evaluating the impact of this kind of training. The one certain con- 
clusion we offer at this time is that we learned a tremendous amount about 
the needs and preferences of teachers as well as of their trainers. To be 
parsimonious, however, we shall present some of the conclusions and obser- 
vations proposed by the PROJECT IMPACT staff in conjunction with our sub- 
ject matter specialists. 

To begin with, It is probably unrealistic, and perhaps even unfair, to 
expect programs of the length, scope, and nature of summer institutes to 
make sweeping, radical, and immediate changes in the participants 1 knowledge, 
attitudes, and teaching practices. This is not to say, however, that by 
exposing teachers, to information, ideas, materials, and methods which they 
are not likely to encounter under other circumstances, that they will not 
benefit. The institutes can complement and supplement, but not by them- 
selves be responsible for, the educational growth that comes through inde- 
pendent study, experience, and graduate instruction. It may well be that 
their short-run impact is most likely to be seen in the transferral of 
selected subject matter concepts and techniques, although institutes commonly 
stress general topics which are expected to have a more profound, longer- 
lasting effect on the participants* thoughts and actions. 

The rationale for conducting institutes was essentially sound, but our 
data have led us to conclude that many institute curricula were too demand- 
ing. That is, we learned from the comments and the actual performance of 
the teachers who cooperated in PROJECT IMPACT, that their past knowledge was 
not readily linked to the new, unfamiliar information and techniques of the 
institute program, in order for learning to be efficient. This is not to 
imply that the participants were not capable of achieving at the high level 
that many of the institutes had set. Some indeed did, and many others 
could have, had the conditions been more favorable. 

Toward this objective, we would urge that the following recommendations, 
coming directly from participants' remarks, be seriously considered for all 
educational development programs. 

1. Elementary as it may sound, we feel compelled to remind developers, 
directors, and trainers, that training programs must always be sensi- 
tive to the interests and needs of the participants. As pointed out 
by our specialists, it would make more sense to determine what poten- 
tial teachers feel they should study than to depend on the proposers' 
impressions of what should be of concern to the teachers. This follows 
directly from our Project 1 data which showed unequivocally that the 
educational objectives and interests of teachers were at variance with 
those of faculty members and directors. Furthermore, attempts on the 
part of the staff and directors to perceive the needs of teachers were 
generally inaccurate. All of this underscores the importance of includ- 
ing participants in all program planning and development stages, 
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Certainly, it should come as no surprise, for after all, teachers are 
closest to children's needs and capabilities, and their emphases and 
orientations are going to be more In deference to children's learning 
habits and preferences. 

2. Teacher-tra i n i ng programs should be relevant to a major and sig- 
nificant part of what participants themselves teach. Topics which are 
esoteric, highly specialized, or too remote from the usual school cur- 
riculum are a waste of time, effort and money. 

3. Training should be practical in orientation. That is, it is impor- 
tant to develop a variety of materials which can be readily used by 
participants in teaching, during training and/or when they return to 
thei r classrooms. 

4. We further suggest that training should be sufficiently flexible 
to allow individual participants to do some independent reading and 
study within the framework of the program. Here again, our subject 
matter specialists concurred that rigid schedules and arbitrary assign- 
ments tend to restrict the advancement and creativity of some teachers 
which in turn, probably inhibits children's motivation to learn and 
curtails their learning progress. Since we agree that teachers should 
encourage flexibility and understanding within their classroom, it 
becomes our responsibility to serve as models. 

A feature of the NDEA summer institutes, most often appreciated by the 
participants and considered exceedingly beneficial, was the opportunity to 
exchange ideas with fellow teachers, institute staff members, and visiting 
specialists. Yet, little attention has been given to this function in the 
planning and execution of institute programs. We feel that a large percent- 
age of participants would welcome such periods of time during which they 
could engage in more systematic dialogues concerning the everyday circum- 
stances of teaching. In fact, it is our opinion that this arrangement 
would serve to reduce frustrations by establishing a receptive atmosphere 
for their expression. The stage is then hopefully set for positive accep- 
tance of the new and the unfamiliar. 

Finally, if any generalization may be fairly drawn from the PROJECT 
IMPACT studies, it is that there was widespread, enthusiastic, acceptance 
of summer institutes. Participants, in particular, felt that attending a 
summer institute was a stimulating and rewarding experience. 
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